Re: [syndication] Re: robots.txt and rss

[Jim Winstead <jimw-yahoo@trainedmonkey.com>]

On Fri, Nov 08, 2002 at 04:03:21PM -0500, Bill Kearney wrote:
> > sometimes the authors notice if you start handing them 403 responses,
> > sometimes not. from yesterday, these hosts got 403 responses from my
> > feeds:
> >
> >   48 cassium.procopia.com
> >   47 csociety.ecn.purdue.edu
> >   72 usersweb1.go-concepts.com
> >   20 wwwcache2-ext.lancs.ac.uk
> >   40 wwwcache3-ext.lancs.ac.uk
> 
> Then how about setting up a sticky trap that opens a session that stays open?
> Or sending them a godzilla-gram of megabytes of pseudo-random XML data?  Havng
> them run out of diskspace or crashing the script is certainly one way to get
> their attention.

i did take julian's suggested approach of spitting out bogus entries for
dead feeds, and these are essentially the leftovers from that. i suspect
some or all are simply long-forgotten cron jobs that will haunt me until
the end of time. that will teach me to provide scraped news feeds. :)

my point was simply that encouraging robots.txt support won't do
anything to solve this class of problem users, which dan seemed to be
suggesting. i'm still skeptical it will solve any class of problem
users, so only see the (albeit marginal) downside of having popular
aggregation tools checking my robots.txt file on the very slim chance
that someday some actual problem user might implement robots.txt
handling correctly.

> This is precisely the sort of reason I've suggested not using HTTP
> errors alone in handling feeds going offline or moving to a new
> location.  Something that indicates, in the feed, that it's being
> moved to a new URL would be one way for anything reading the feed to
> understand where to go instead.
> 
> But yes, supporting genuine HTTP error codes IS important.

yes, and i'm glad to see advocacy of handling HTTP error codes and
caching-related headers on the rise (and will catch up on the server
side for my own feeds some day).

support of the skipHours and other similar methods are great, too.

jim