Digiwar - the Yeep-blog

I have made very good progress on my news aggregator using the Google Reader back-end. It’s in a state where it’s useful enough for dialy use. So that’s exactly what I’m going to do. Also, with a few minor adjustments, I’m going to release it as the first alpha version. I’m really curious as to if people will use it in this state.

Back to Visual Studio for me!

[Last played: Marilyn Manson - Tainted Love]

The Google API for Google Reader has not been released. At least officially. I found a real good head-up at Niall Kennedy’s weblog. The information there is more then enough to get you started. The information in the main article is not complete though. You can find out some more tidbits in the comments and I also found some more stuff while working on FeedME. It hadn’t occured to me how easy it actually is to get some information about the API. I figured, it’s a web application, so all the interesting communication happens at the server itself, no way I can get to that. I had even downloaded the javascript file Google Reader uses for it’s work when it occured to me that, yes, this is a web application. It’s a web 2.0 (I really hate that term, but now everybody should know what I mean) application. It uses Ajax, which means that all communication with the server backend originates from my own machine! Add Fiddler and you’ve got a neat protocol-sniffing-setup
I’m thinking about doing a page on the Google Reader API to put all the info I have at one location.

Anyways this is not about the Google Reader API, at least not in depth. What I am going to write about is the HttpWebRequest, WebResponse and the Uri classes of the .NET framework 2.0.
These were the classes that I wanted to use for communicating with Google. The are built for that purpose! In fact, they worked perfectly for the first API I incorperated, the call to download your subscription list. After I had that working and had a really basic GUI I was ready to move on to the next thing: Downloading the articles for the feeds I’m subscribed to.
The Google Reader API is a REST API (or at least based on some REST principles). This means is uses simple HTTP GET and HTTP POST commands to communicate with XML. Okay, so this also covers SOAP, but the thing with REST is that it doesn’t have predefined XML to use. It’s all up to the implementation. Whereas SOAP has everything standardized except the data itself.

Getting your subscription list is just sending a GET at a specific URL, let’s say: “http://www.google.com/reader/api/subscriptions/” (This isn’t the real URL, just an example). You’ll then get some XML back that describes your subscriptions. Simple to do with HttpWebRequest.
To get the articles of a specific feed, you need to add the URL of the feed to the URL. An example (also not real): “http://www.google.com/reader/api/feed/http://www.digiwar.com/feeds/”
This would give you the articles from this website that Google have in their Reader database. Now here I ran into a problem.

The HttpWebRequest class takes, or converts your string into, a Uri object. The Uri object is a real useful class that does all kinds of useful stuff to the URL you supply. Like checking if it’s a valid URL, removing contradicting directory paths or removing those pesky doubles slashes some people put in their URLs by accident. Do you see the problem already? If not, I’ll give you 5 minutes to think about it.

Take your time.

You got it? Good. So you aggree with me that I was really frustrated that my HttpWebRequests ended up going to “http://www.google.com/reader/api/feed/http:/www.digiwar.com/feeds/” instead of the correct URL. Nothing I could do would help. I tried double-double slashes, tried escaping them, tried encoding them as “%2F”. Nothing worked. I did a lot of Googling on the subject, inquired in newsgroups. Nowhere did I find an answer. So what did I do? I wrote my own HttpRequest, HttpResponse and Url classes.

The Url class was really simple. It just houses a few strings and does some parsing in the constructor. It does no validation whatsoever.
The HttpRequest and the HttpResponse class where a bit harder. For them I P/Invoked a lot of WinInet functions. One important thing learned here: If you ever need the P/Invoke signature of a Win32 function, or at least a good pointer in the right direction, just google for “dllimport “. Also visit pinvoke.net, it’s really useful.

And ofcourse, after I had everything up and running (beautifully I might add) some one found an answer for me. Apparently if you encode the second double slash as ‘\u2215′. So you’ll end up with “http://www.google.com/reader/api/feed/http:/\u2215www.digiwar.com/feeds/” and this seems to work. I still have to test it in a realy world scenario (such as my program), but I’ve just invested a lot of time in my custom classes and it’s working perfectly, so why switch?
I might switch back to the .NET classes sometime in the future, but for now I’m content with how it is.

[Now playing: The Kovenant - Jihad]

So I took the task upon myself to write a Windows application that allows me to sync with Google Reader and allows me to read my feeds I have in Google Reader. This is not a post for plugging that application though. This is a post, or probably one of a series of posts, about the things I learned while programming the application.

But to spend just a few words on the application: I’m going to call it “FeedME”. I had thought up that name probably two years ago when me and a colleague were already considering writing a feed reader. The project never got really far, but the name stuck.

So far the application allows you to log in, retrieve the feeds your subscribed to and then the unread articles from those feeds. The articles are show in a real simple newspaper format in an embedded Internet Explorer control. That’s about it.
The biggest hurdles I crossed so far? Downloading the feeds from Google (getting the subscription list is easy, getting the articles of a particular feed is much harder, more on that later) and logging into Google.

Logging into the Google Reader is complex, but not really that hard when you think about it. The thing is you can’t just send a username and password combination to a webservic. At least not that I know of. No, you have to use cookies and then, somehow, find out the Google SID for the user. I will explain this in detail later!

So, expect some articles about the problems I encountered and how I solved them (or worked around them) in the future. And if I get it into a workable state, I will release FeedME for everyone to use. Free of charge!

Right now I’m gonna see if I can come up with a nice way to mark a seperate article as ‘read’ and ‘unread’ and how to tell Google that and how to show that in my application (no support for read articles yet).

[Last played: Static-X - Trance is the motion]

I think I’ve been using feeds (in one way or another) for around 5 years now.
It all started with KlipFolio. I saw a review of it on some website and really liked it. The whole idea of reading websites without actually having to go to the websites and see if anything new had been posted really appealed to me. I totally dug it. I didn’t need to go the the news, the news would come to me. Lovely!

It took some time for my favourite websites to add a klip of their own, but in time more and more allowed me to read them with this nifty program. And some websites that didn’t have a ‘klip’, did have an RSS feed, or whatever the hell that was. There was a plugin for KlipFolio that also allowed me to view RSS feeds. This really increased the number of websites I could read in KlipFolio. So then I ran into a problem. Although the program is really usefull, it’s not that usefull if you want to read more then a handfull of websites. I had more klips then I had screenspace to view them on. So I tried looking further.
Using my newfound knowledge of RSS I had a look at some RSS browsers. I tried a few, but finally settled on one of the first FeedDemon beta’s. I had it running for quite a while (the early versions limited the number of times the program could startup to 30, later versions only allowed the program to run unregistered for 30 days).
I liked FeedDemon, but I hated that I couldn’t watch all my feeds at once. You could organize your feeds into categories, but you could only watch one category at a time. And if I recall correctly, FeedDemon would also only update the feeds in the active category. So time to look further.

Then I found RSS Bandit. A free, opensource feed reader that did almost everything I wanted. Almost, because there were two things that I really missed.
The first being able to synchronize between computers. I read feeds at work and then also at home. So I want the feeds I read at work, not to show up as unread when I’m at home. RSS Bandit added that functionality later on, but it never worked flawlessly for me.
The other thing I really missed was that the articles in a feed have a limited lifespan. You can define how much articles you want to have show up in a feed at one time. Usually this is between 10 and 20 articles. As soon as the 21st article is added to the feed, the 1st falls out. So if you have a high-traffic news site like Slashdot and you don’t read their feed for two days, you miss out on a lot. I ’solved’ this problem by keeping RSS Bandit running at work at all times. That way RSS Bandit itself would store the articles for me to read later on. But the situation was less then ideal.

On to the online feed readers, oh pardon me: news aggregators! There were a bunch, but not many are not any good. The best ones were BlogLines and NewsGator. I went for NewsGator and I’ve been using that for about a year and a half now or something. I like it. It’s the best way of reading news so far, but still it has a real annoying issue. Although NewsGator stores all the articles from a newssite for quite some time, it only keeps read/unread information about the articles currently in the feed. So although I can read through a backlog of Slashdot articles if I spend the weekend away, I do have to keep track of which I read and which not. Now this isn’t a real big issue, but after a while it really started nagging me.

So what to do next? Oh, hey look! Google released a new online new aggregator. Too bad it sucks. That is, the interface they decided to give you sucks, at least in my opinion. But this is 2006 and Web 2.0 is the buzzword of the day! So why not combine the back-end of Google, with my own front-end? Yay! Project time!!!!

[Last played: Static-X - Trance is the motion]

World of Warcraft is the biggest MMORPG in the world. Last week they announced they had 6 million subscribers. A ‘Massively Multiplayer’ game markets itself as being able to play a game with hundreds or thousands of other players. But how much truth is there to that?
World of Warcraft added a new high-level dungeon a few months ago, but instead of making it immediatly accessable the players on each server had to put in some effort to get them open. It required all players to harvest certain goods and then a long, difficult and expensive quest chain to get the ‘key’ to the gates of the dungeon. The opening of the gates are a one-time event, a spectacle that will occur only once per server. So naturally everyone wants to be there to witness it. This means that a lot of players (several hundred) will be in one place at the same time. This causes massive lag, although I suspect much of the percepted ‘lag’ is just the client having framerate problems, and massive disconnects. I’ve spend more time logging back in and waiting to be able to enter the world then actually participating in the event.
So how ‘massive’ is it when you can’t even witness an event because you try to “play online with hundreds of other people from around the world”?

[Last played: Within Temptation - Our farewell]