Re: [syndication] robots.txt and rss

["Bill Kearney" <wkearney99@hotmail.com>]

> I've taken to using a robots.txt file to ban (polite) web crawlers from my
> site because I've got memory problems and can't handle them thrashing my box.
> Does this mean that RSS fetchers will no longer be fetching my rss files?
>
> Should I explicitly allow my RSS files in my robots file?

No, RSS tools don't look at the robots.txt files.  The RSS files are, most of
the time, served as if they were just regular web pages.  So they'll neither
notice nor use anything in the robots.txt file.

-Bill Kearney