RE: [syndication] Re: Robot Discovery ) OT Banning Aggregators

["Alis Marsden" <alis@contentrelations.com>]

Hello all,

Please excuse me for coming in on this discussion when I've not been
following it.

I think it is possible that some content providers may wish to ban certain
headline aggregators in the future.  If a Robots.txt style solution is up
for discussion, I'd like to throw that thought in, in case it is relevant to
what you are discussing.

I'll explain why I've thought about this.  Please accept that this is not
something I've thought about well, just some random ideas I'd like to get to
the group :)

As we know, headline aggregators use the data they aggregate in a number of
ways - news portals, news search engines, providing newsfeeds made up from
various headline providers to web or intranet sites and so forth.

One of the other things we see them doing is providing intelligent
monitoring services.  Moreover (http://www.moreover.com/) would be an
example of a headline aggregator that provides such services.  If you read
their white papers you will see they don't just aggregate news headlines,
they do discussion boards, newsgroups and so on.  So far as I know you can
request new sources and give them keywords you want to 'watch'.

Other companies that don't have any obvious connections to RSS/headline
aggregation offer the same kind of services.  For example Envisional
(http://www.envisional.com/).  Paola Di Maio of Content-Wire wrote an
article about them earlier this year:
http://www.content-wire.com/Home/Index.cfm?ccs=86&cs=440.  Webclipping
(http://www.webclipping.com/) is another example.

I'm not saying intelligent monitoring services are a bad thing but they are
not always a good thing in my opinion.  For example I have become aware,
over the last two years, of how legal people use the Internet in order to
research and monitor key people from companies their clients have disputes
with.  If I was being monitored like that I wouldn't like it.  I know that
posts made to discussion lists like Yahoo!Groups are not private.   Still if
I was a maintainer of a discussion board, I think I might like to be able to
say no to some headline aggregators, because of concerns I might have about
why they aggregate.

Currently I think most content providers think RSS and headline aggregation
are good things.  When I talk to people like discussion board maintainers,
they want their users to be able to use headline reading software and want
the traffic syndicating their headlines brings.  Indeed, It is a good thing
in my opinion.

However I am concerned that they may start to see people like Moreover
offering monitoring services and think 'I don't want my website users
monitored!' and so headline syndication is not something I'm going to do.
If there was a way to say yes to some types of aggregators and no to others
that might be useful.  Is this an idea that might tie into a Robots.txt for
RSS concept?

Alis Marsden