Re: [syndication] Re: robots.txt and rss

["Bill Kearney" <wkearney99@hotmail.com>]

> You might want to restrict certain (disfunctional, annoying etc)
> user-agents, eg. if they poll impolitely often.

If they're that dysfunctional they're certainly not going to respect/understand
how to parse out a robots.txt file.

Asking the readers to engage in extra behavior to avoid making consumption
mistakes?  How about asking the readers to just start being mindful of the data
and optionals /already available/ in the standards?  It's a lot more likely a
programmer is going to add other XML handling routines that it is to pull
another text file and slog through it.

If dealing with abusive consumption is the issue then tackle it at the server
side.  Apache has fine tools to aid the process.

-Bill Kearney