Digiwar - the Yeep-blog

I have made very good progress on my news aggregator using the Google Reader back-end. It’s in a state where it’s useful enough for dialy use. So that’s exactly what I’m going to do. Also, with a few minor adjustments, I’m going to release it as the first alpha version. I’m really curious as to if people will use it in this state.

Back to Visual Studio for me!

[Last played: Marilyn Manson - Tainted Love]

So I took the task upon myself to write a Windows application that allows me to sync with Google Reader and allows me to read my feeds I have in Google Reader. This is not a post for plugging that application though. This is a post, or probably one of a series of posts, about the things I learned while programming the application.

But to spend just a few words on the application: I’m going to call it “FeedME”. I had thought up that name probably two years ago when me and a colleague were already considering writing a feed reader. The project never got really far, but the name stuck.

So far the application allows you to log in, retrieve the feeds your subscribed to and then the unread articles from those feeds. The articles are show in a real simple newspaper format in an embedded Internet Explorer control. That’s about it.
The biggest hurdles I crossed so far? Downloading the feeds from Google (getting the subscription list is easy, getting the articles of a particular feed is much harder, more on that later) and logging into Google.

Logging into the Google Reader is complex, but not really that hard when you think about it. The thing is you can’t just send a username and password combination to a webservic. At least not that I know of. No, you have to use cookies and then, somehow, find out the Google SID for the user. I will explain this in detail later!

So, expect some articles about the problems I encountered and how I solved them (or worked around them) in the future. And if I get it into a workable state, I will release FeedME for everyone to use. Free of charge!

Right now I’m gonna see if I can come up with a nice way to mark a seperate article as ‘read’ and ‘unread’ and how to tell Google that and how to show that in my application (no support for read articles yet).

[Last played: Static-X - Trance is the motion]

I think I’ve been using feeds (in one way or another) for around 5 years now.
It all started with KlipFolio. I saw a review of it on some website and really liked it. The whole idea of reading websites without actually having to go to the websites and see if anything new had been posted really appealed to me. I totally dug it. I didn’t need to go the the news, the news would come to me. Lovely!

It took some time for my favourite websites to add a klip of their own, but in time more and more allowed me to read them with this nifty program. And some websites that didn’t have a ‘klip’, did have an RSS feed, or whatever the hell that was. There was a plugin for KlipFolio that also allowed me to view RSS feeds. This really increased the number of websites I could read in KlipFolio. So then I ran into a problem. Although the program is really usefull, it’s not that usefull if you want to read more then a handfull of websites. I had more klips then I had screenspace to view them on. So I tried looking further.
Using my newfound knowledge of RSS I had a look at some RSS browsers. I tried a few, but finally settled on one of the first FeedDemon beta’s. I had it running for quite a while (the early versions limited the number of times the program could startup to 30, later versions only allowed the program to run unregistered for 30 days).
I liked FeedDemon, but I hated that I couldn’t watch all my feeds at once. You could organize your feeds into categories, but you could only watch one category at a time. And if I recall correctly, FeedDemon would also only update the feeds in the active category. So time to look further.

Then I found RSS Bandit. A free, opensource feed reader that did almost everything I wanted. Almost, because there were two things that I really missed.
The first being able to synchronize between computers. I read feeds at work and then also at home. So I want the feeds I read at work, not to show up as unread when I’m at home. RSS Bandit added that functionality later on, but it never worked flawlessly for me.
The other thing I really missed was that the articles in a feed have a limited lifespan. You can define how much articles you want to have show up in a feed at one time. Usually this is between 10 and 20 articles. As soon as the 21st article is added to the feed, the 1st falls out. So if you have a high-traffic news site like Slashdot and you don’t read their feed for two days, you miss out on a lot. I ’solved’ this problem by keeping RSS Bandit running at work at all times. That way RSS Bandit itself would store the articles for me to read later on. But the situation was less then ideal.

On to the online feed readers, oh pardon me: news aggregators! There were a bunch, but not many are not any good. The best ones were BlogLines and NewsGator. I went for NewsGator and I’ve been using that for about a year and a half now or something. I like it. It’s the best way of reading news so far, but still it has a real annoying issue. Although NewsGator stores all the articles from a newssite for quite some time, it only keeps read/unread information about the articles currently in the feed. So although I can read through a backlog of Slashdot articles if I spend the weekend away, I do have to keep track of which I read and which not. Now this isn’t a real big issue, but after a while it really started nagging me.

So what to do next? Oh, hey look! Google released a new online new aggregator. Too bad it sucks. That is, the interface they decided to give you sucks, at least in my opinion. But this is 2006 and Web 2.0 is the buzzword of the day! So why not combine the back-end of Google, with my own front-end? Yay! Project time!!!!

[Last played: Static-X - Trance is the motion]

I’ve been using RSS feeds for the past 9 months or so and I’ve seen some quite nice improvements. Not in RSS itself, but it the way sites use it.

Take for instance Slashdot. they used to have a feed that contained just an excerpt of the article. Since a few weeks now the feed contains the full article text and still no ads. I love it!!

Then there is Blabbermouth, my source of Metal news. They used to be without RSS feed and I had to lookup the site everyday to see if anything interesting was posted. I was about the mail their webmaster asking for RSS support when all of a sudden they had redone their site and had included an RSS feed! The feed does not contain the full article text, but it’s also not an automated “first paragraph only” feed. They actually use a seperate excerp that sums up the article. This is great, because it prevents items like “John done informed us of the following: …”.

On a more personal note I implemented an RSS feed as part of the logging of an automated process I wrote the scripts for. This way I can keep an eye on the process as well. I have a normal error feed and a debug feed, which gives me some debug info. It’s been useful already!

How could I have missed this? Sure, I haven’t been following Jabber closely for a while now, but this is really interesting. Jabber has a Publisher/Subscriber (PubSub) JEP (Jabber Extension Protocol) and there is already a site using that JEP to give you RSS/ATOM-feeds and Newsgroups in your Jabber client. How cool is that? You basically get an IM when a weblog has been updated and you can then view it inside your IM client.
In this document they even describe how they do it. Very interesting.

On his informative site for new RSS users, Dave Winer proposed the readers of his site to start writing reviews of the reader programs they use. The idea is that users will have a much more useful perspective for potential users then some marketing guy or lead-programmer who writes up a feature list for his product. I think this is a very good idea. And I urge everyone who wants to, to write a review for his favorite reader.

Dave Winer has a very interesting blog-entry up on his new RSS for newbies website. He has arguments for the webmaster that still wonders if RSS would be a benefit, or a loss of traffic/income. If you are a webmaster and need convincing RSS feeds are good for your site, read the entry.

Okay, this may be a bit exaggerated, but it’s essentially what this Wired article means to say. Dave Winer also commented on it. He had the same idea I had when I read the article: combine RSS with Bit-Torrent. Peer-to-peer RSS networks may be the future. Or maybe those damn RSS reader developers should get a clue and support conditional GETs and honor the skiphours/skipdays tags in RSS without a means to overule it.

In a recent post I explained a simple idea for using RSS as a concert notification system. Someone linked to my article on another blog and a comment posted on that blog pointed to Tupamaros Online, an Italian band that actually have a concert notification through RSS since 2001. It’s not exactly like I meant it, but it is definitly the first band I’ve seen that does this. I just hope more bands get a clue, as this would definitly enhance my concert awareness.

Although I’ve seen it being linked from a zillion news sites and blogs, and although the chances are slim you haven’t seen it already, I’m still going to give you the link to The XML Files: All About Blogs and RSS.

Yesterday I talked about more uses for RSS then just headlines. Well today I found two more, real-life examples of it’s usefulness.
First there is CVS2RSS a Perl script that will take the changelog of a CVS checkin and create an RSS feed out of it.
Secondly there is MailFeed that allows you to log on to an IMAP, POP3 or NNTP server and will then generate an RSS feed based on the new messages.

I think these two projects show that there are more people out there that think RSS can do so much more then just push around headlines.

RSS is mainly used for one of two purposes as far as I know:

1. New entries is a weblog
2. Headlines for a news site

But why stop there? RSS can do so much more. Especially with the extendability of RSS. Granted the tools would need to know how to handle them, but with this you could create specialized RSS feeds with specialized RSS tools. Some examples of what I would find useful.

Concert agenda
I subscribe to an RSS feed that lists concerts when they are announced, then a month before they’re supposed to happen and then a day or 2 before the happening. This would not require additions to the RSS feed, just smart software.

Personal agenda
Make me a website where I can write down my appointments. Then have it create a feed that I retrieve at the start of every day (or perhaps twice, at the end and at the start) and I’ll know exactly what appointments I have that day. It’s not real-time with pop-ups like Exchange/Outlook, but it’s enough for home-use (and a lot cheaper too).
Still no additions, just smart software.

Event notifications
I do work for ISPs mainly and we use a Trouble Ticket System to handle problems on our platform. Why not have that system generate an RSS feed for the different kinds of problems we have. Add some sort of ‘recall’ mechanism in the RSS feed and use a reader that supports it and it could even remove the notifications about problems that have been solved.
This wouldn’t replace the system, but it would be a very useful tool for people like me who don’t need to create or edit the tickets, but just need to keep an eye out for something that concerns me.

I see RSS more as a non-real-time notitication mechanism rather then a headline aggregation mechanism. It can do so much more. I recently heard about someone who thought about combining RSS with BitTorrent so he could receive download instrcutions at night and have his computer download stuff while he was sleeping. Maybe you could combine RSS with Jabber (or XMPP) and have the newssite send a sort of broadcast notification when a feed has been updated. No more polling required.
Ofcourse you could just say: “Why not just send the item itself with Jabber”, but it’s just an idea I’m playing with, not sure it it’s useful. But who knows