Re: [syndication] Re: robots.txt and rss

[Dan Brickley <daniel.brickley@bristol.ac.uk>]

On Fri, 8 Nov 2002, Ben Hammersley wrote:

> On Friday, November 8, 2002, at 04:02  pm, Dave Winer wrote:
>
> >> opinion:  RSS aggregators probably *should* respect robots.txt.
> >
> > I agree. If someone were to post a BDG, like the one [1] Simon Fell
> > posted for Etags, I would put together support for it in Radio
> > (assuming it was easy, which I assume it is).
> >
>
> Why would you have an rss feed, and an robots.txt that restricts using
> it? Just don't publish an RSS feed.

You might want to restrict certain (disfunctional, annoying etc)
user-agents, eg. if they poll impolitely often.

Dan