Sunday, September 12, 2010

-O fun || Designing for Hackability

My little side project Streamie was designed to be hackable, as in easy for somebody else to make changes on, from the ground up.
Besides the ability to run Streamie against your own github repository without running your own backend server, there are several other design decisions in the code that are supposed to make the source as approachable as possible.
I'm joining Google next month, so the obvious choice for a client framework would have been to use closure. I decided to go with jQuery instead because everybody already knows that one already.

app.js
When you want to start digging into Streamie, start by reading app.js. In this file you see two big static arrays:
streamPlugins: This is a list of "plugins" which are run on every single tweet and which can transform the tweet in any way. Currently all stream plugins are defined in streamplugins.js. If you want to extend Streamie's tweet view, like e.g. translating all tweets, this is where you want to add your own plugin.
initPlugins: This is a list of plugins which are run as soon as the connection to the backend system has been established and Streamie is basically "running".

The REST API
Twitter exposes an extensive REST API to interact with it's service. Using these APIs requires doing oAuth. Doing anything with oAuth sucks. The good thing is that Streamie provides a proxy for the REST API that does all the oAuth stuff for you. This means you can use the API just as it is documented. E.G. this is how you favorite something in Streamie. Because the main channel from Twitter to Streamie is the user stream API, you don't really have to worry about rate limits.

Tweets, Direct Messages, Retweets
Coming from the Twitter API Tweets, Direct Messages, Retweets look quite different. Streamie normalizes them to be all specialized Tweets. This makes working with them quite easy. Streamie is using the "Stream" metaphor extensively. That is why your direct messages appear in the main stream. The buttons in the navigation only activate CSS classes which filter out certain tweets.

Start up time
Streamie's startup time can be quite long (as in a couple seconds). This is mostly due to the time it takes to load initial data from twitter. (See here if you are curious how that works). In development you can append ?cache to the URL and these API calls will use sessionStorage caching which makes loading almost instant.

Learnings
Overall Streamie's software design is quite simplistic. Right now it feels like there might be a need to abstract some stuff a little more. There will likely be classes for "status inputs", custom events and maybe even tweets :)
Do you have questions or suggestions about how Streamie works or is supposed to work?

8 comments:

jups42 said...

I think the stream functionality is ideal for searching in twitter too. I'd love if it could be the next thing to work on.

Malte said...

Would be cool yeah. I'm not sure whether it is really feasible. The streaming API only allows access to samples of search data http://dev.twitter.com/pages/streaming_api_methods#statuses-sample

Malte said...

Oh, I see that user streams support the track-searches, so this should be quite possible.

You have to reconnect upon changing the track words. Adding this to Streamie should be extremely simple, actually. One would have to see how the tweets that came from tracking can be differentiated so that they can be tagged with a class for filtering.

jerome etienne said...

http://github.com/cramforce/streamie/tree/master/public/prototype/ this one is still used ? or just obsolete code ? if so can you remove it ? just trying to make the code easier to understand

Jerome Etienne said...

it would be cool if you could you add more on the link plugins. http://github.com/cramforce/streamie/blob/18dc3093cd81fb8676d5d6bcfc85d49fae9773b2/public/lib/stream/linkplugins.js

Jerome Etienne said...

i think it would be easier to add/merge plugins (of all 3 kinds init/stream/link) if it wasnt packed in a single file... currently adding a plugin will likely result in a merge conflict.

maybe something like the folders hierarchy of unix /etc/init.d but adapted to js/browser...

Jerome Etienne said...

i think it would be easier to add/merge plugins (of all 3 kinds init/stream/link) if it wasnt packed in a single file... currently adding a plugin will likely result in a merge conflict.

maybe something like the folders hierarchy of unix /etc/init.d but adapted to js/browser...

Jerome Etienne said...

btw im making a lot of comments... just suggesting here. no pressure