robots.txt links

["Bill Kearney" <wkearney@syndic8.com>]

http://www.robotstxt.org/wc/robots.html

And don't forget, the archive can help if old pages have gone offline, or to
find past revisions:
http://web.archive.org/web/*/http://info.webcrawler.com/mak/projects/robots/
http://web.archive.org/web/20010711193857/www.robotstxt.org/wc/

It would be /very/ interesting to see how various tools handled unexpected data
showing up in the robots.txt file.  Let's get a grip on the "but we'd break
legacy hacks" risk before this goes too far.

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

-Bill Kearney