Re: [syndication] site-wide metadata discovery

[Klaus Johannes Rusch <KlausRusch@atmedia.net>]

Bill Kearney wrote:

> Perhaps we can all wander forth and find the reasons, pro and con, for using
> extensions to the robots.txt file.  Let's see if we can dig up circumstances or
> use cases that demonstrate why or why not doing this is workable.  I'd hate to
> see us latch onto the idea only to have the experts trash it and open up a whole
> permathread of arguments on it.

One concern is that search engines, at least traditionally, have implemented size
limits for robots.txt and excluded sites with overly large robots.txt files. Not
sure if that restriction still applies (most robots requesting robots.txt will not
use the additional information though so robots.txt is not an ideal place for this)

--
Klaus Johannes Rusch
KlausRusch@atmedia.net
http://www.atmedia.net/KlausRusch/