Re: [syndication] An idea for RSS distribution - push it through Jabber!

[Dion Loy <dion-yhoo@loy-fu.com>]

Hi Jeff,

I was actually thinking through the same things yesterday. I was about
to start the design and class structure today.

My take was slightly different though, perhaps we can merge the two
efforts somehow, or bounce some ideas off each other ...

I was about to code up a Jabber agent, that would register at a
specific address, ie headlines@jabber.com.

Users logged into Jabber that wished to receive headline updates,
would send instant messages to this address to register, ie:

1. 'subscribe Userland'
2. 'subscribe http://www.cnn.com/cnn.rss'

Now these would be 'fed' into the headline agent through another
publically available server (probably in XML-RPC/SOAP format, maybe
with the Blogger API ??).

You would register your RSS file under a 'keyword' (in example (1)
above, it would have been 'Userland').  The server will store your RSS
file and disseminate updates to all those subscribed to that
'keyword'.

If the RSS is not registered, then the user would have to use method
(2), which would poll and aggregate RSS files for distribution.

We can scale this up by putting multiple datastores / request handlers
on separate machines even on different networks:

                     [jabber client]
                             |
                             |
                     [Jabber.com server]
                             |
                             |
                     [headline agent]
                            /|\
                           / | \
                          /  |  \
                         /   |   \
                       [1]  [2]  [3]

Where [1], [2], [3] are the aggregator and data store servers that
hold the RSS.

In fact, the 'return' message, might not even have to go back through
the headline agent to reduce the bottleneck, we could just IM the
results directly from [1], [2], [3] to the [jabber client] (it would
come from a different address).

News publishers would send their updates to one [1], [2], [3] which
would communicate between themselves to keep their data stores up to
date.  These data stores would also be responsible for polling for
updates for specific RSS files requested by users that are not sent in
by the news providers themselves.

Are we thinking along the same lines?  I was about to set up a domain
name today and start writing some documentation for this idea. Or were
you thinking of using Headline Viewer as the Jabber client?  I suppose
with the backend in place, it really wouldn't matter or care what
client it was talking to as long as it's Jabberese =).

Thursday, August 23, 2001, 7:07:45 AM, you wrote:

JB> I have been thinking about the problem of RSS clients overloading
JB> news sites such as newsisfree.com.

JB> Then I read all about Dave Winer's work with XML-RPC and Jabber:

        
JB> http://scriptingnews.userland.com/backissues/2001/08/22#tunnelingXmlrpcW
JB> ithJabber

JB> And then I read the following message:

JB>         http://wmf.editthispage.com/discuss/msgReader$5270?mode=day

JB> And when I woke up this morning I realized that we may have an 
JB> interesting solution. To wit:

JB>         What if we use Jabber as our distribution vehicle?

JB> It is based on XML, it is widely available, and the processing 
JB> overhead seems to be pretty low.

JB> Basically we would create a single "Newsfeeds" conference.

JB> Providers of syndicated material (or third-party re-posters) would
JB> then post new messages into the conference. Each message would be a
JB> complete RSS file. We could create multiple conferences to reduce
JB> the load on the client, but this is a decision we can make later.

JB> Clients would subscribe to the conference, and would be notified
JB> when new news is available.

JB> I have not thought this through all the way, but I think it could
JB> actually work. One interesting aspect of this is that it would
JB> force the use of valid XML (you can stop cheering now, Morbus :-).
JB> It would obviate the need for clients to poll for updates. And it
JB> would give us something akin to a real-time news feed, which would
JB> be very, very cool.

JB> We do gain a new single point of failure. If it turns out that
JB> the servers at jabber.com are not reliable enough, we could consider
JB> running our own server(s).

JB> How does this sound? Is it crazy enough to work? Again I will 
JB> repeat my usual offer -- if someone wants to do the server side
JB> stuff I will add experimental support for this to Headline Viewer.
JB> Do we have a news provider ready to step forward and start posting
JB> RSS to Jabber?

JB> Ideas, feedback, etc?

JB> Jeff;


 

JB> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/