Aggregating and displaying feeds

[Julian Bond <julian@netmarketseurope.com>]

I started a discussion on another list but it seems appropriate here as
well.

Let's say you're writing YAFA (Yet Another Feed Aggregator!). I think
there's some conceptual design thought about how you display the results
from multiple feeds. I want to try and get away from displaying the
results sorted by feed and sort them by relevance, age and subject
instead.

The first take on this is that any one feed is pretty much limited to a
single subject (excepting blogs of course, although individual bloggers
tend to circle around specific subjects). Given this, a first cut is to
group feeds into bundles something like the way NewsisFree allows you to
define a "Page" with multiple feeds on it. If you store the items in a
database, unique on <title> and <link> and with a timestamp, it's pretty
easy to pull them out in LIFO order. 

If you allow Admins and Users to define their own bundles, you start to
get into a many to many to many situation which is not hard in SQL but
produces lots of linking tables and Joins.  

So far, so do-able. It's a SMOP! To put some meat on this, I could
define a bundle on Motorcycle Racing which was a Moreover search + a
feed from Motorcycle News, motorcycle.com, amasuperbike.com, MotoGP.com
etc.

Where it gets interesting is when you start to think about categories
and rating. And how the rating affects the visibility and priority of a
particular <item>. It might be possible to get users to rate a
particular feed and possibly even a source. But individual items is too
much work for them. And source is often not easily available. I think
the ideal is to just use emergent behaviour to drive the system rather
than relying too heavily on active input. If you can aggregate the click
throughs and cross match it with the user, feed and source, there's some
powerful rating to be done behind the scenes. If either the feeds or the
original web pages had a bit more metadata in them (like Dublin core),
there's another set of data to be mined.

Where I'm going with this is of course, "The Daily Me". I'm curious to
know just how close we can get to this, using the data that's available
right now and without introducing new elements in the standards or a
huge push to get people to produce richer feeds. 

-- 
Julian Bond eMail: julian@netmarketseurope.com
HomeURL: http://www.shockwav.demon.co.uk/ 
WorkURL: http://www.netmarketseurope.com/
WebLog: http://roguemoon.manilasites.com/
M: +44 (0)77 5907 2173  T: +44 (0)20 7420 4363  
ICQ:33679668 tag:So many words, so little time