Collaborative Content Delivery

["Werner Vogels" <vogels@cs.cornell.edu>]

[I sent this message earlier to the decentralized list, but also post it
here instead of cross-posting. We are very interested in getting
comments/feedback/help from the people on this list]

Somewhere in October last year there was a brief discussion on the
decentralization list on push-vs-pull for publishing news items. Many
people were very positive about the idea of using decentralized
techniques for the delivery of news items, especially for when the
publishing system comes under emergency stress. We have picked up on
that idea and mocked up a quick prototype using a toolkit that we have
built for scalable distributed management, and the initial result looked
good enough to convert this into a real project/product.

The system we hope to distribute by this summer will do Collaborative
Content Delivery, that is use the collaborative power of the internet to
weave a publish/subscribe infrastructure for the real-time delivery of
news items. It is our believe that we can build a system that can
deliver news within tens of seconds to tens or hundreds of thousands of
subscribers, reliably, based on p2p technologies. With secure
authentication of the publishers as a basic requirement.

The infrastructure is based on Astrolabe, a toolkit that uses epidemic
protocols and information aggregation & fusion techniques to maintain
distributed state in an ultra-scalable fashion. We'll use it to manage
the subscriptions and use subscription aggregation  to do subscription
based news item forwarding.

The infrastructure part is not what I am worried about most, it is where
our expertise is and although a lot of work needs to be done, I am sure
we will get there. Almost the biggest problem to we have to solve in
this part of the project is the auto-configuration & organization part,
especially with an eye on the secure aspects of the system. If we want
to have a targeted system for download in 6 months with which people can
subscribe to a limited set of news sources, what is the minimum
configuration we need to have them do after download...

The part that does worry me is the usability aspect. As I have learned
in the past months about syndication, I understand we are complete
novices in that area. We need to do a lot of work to make sure the
system can meet the expectations of the community, as this is after all
what drives the success of a p2p application.

I would very much like to invite the help from the community here. I
would be very much interested in hearing what people think what would
make a system like this useable? How could we exploit in the
infrastructure work domain specific knowledge about news items
packaging, etc.? What are the services publishers and subscriber would
need?

Any comments or actual help would very much appreciated.

Some more background information can be found at
http://www.cs.cornell.edu/vogels/NewsWire. There is high-level
description of the system in a paper I wrote last year, where certainly
at the end it show my ignorance wrt to rss/syndication, etc. There are
also more links to the Astrolabe technology and the draft of a slide set
for a presentation at this weeks internet2 P2P workshop

--
Werner