Saturday, December 11, 2010

The Slowest JavaScript Application on Earth

While writing an article about profiling browser-based JavaScript applications I thought it might be fun to have the worst possible example to make it better step-by-step.

Thus I created this VaporJS based Github repo which will accept only the worst offenders in JavaScript performance best practices. Pull requests that make it slower or add cool new bad performing features are very much appreciated.
Ideas
  • Do some blocking Ajax calls.
  • I think we need jQuery selectors that won't make it through querySelectorAll.
  • Some legacy Prototype.js code to slow down Array operations can't be bad.
  • Add some CSS that will make rendering really hard (The * selector can help).
  • Hint: Writing to localStorage blocks the UI thread in Chrome and Firefox.
  • When you load ressources, make sure you always bust the cache so it doesn't get in your way.
  • A little flash movie with the right wmode can have devastating effects.
If you are scared to contribute because future employers might find the code and not hire you, we provide a wikileaks-style-highly-anonymous way to send in patches: @-messages a link to @cramforce.

Important: Thanks for all the awesome contributions. Please make sure the changes are realistic (as in: some people would actually put them in because they don't know better) and add features.

Monday, October 25, 2010

Eliminating the Need for Flash

HTML5 and related technologies are getting more powerful every day and for me as a web developer it feels good that it becomes possible to build event more things that would have been the domain of "Flash Developers" a couple years ago (if not event cooler stuff). And, actually, I was using Flash myself to make things work back in those days.

Still there are two things that would still make me pick Flash as the technology of choice and I don't like that, so we should fix it!

1. Fullscreen
Fullscreen is awesome. Even Steve Jobs thinks so. I personally think we should do something really simple to fix this: window.open('/my/path', '_fullscreen')
window.open already has all the security restrictions in place that we expect from a modern browser with a built in popup blocker. There might need to be extra restrictions with respect to keyboard events (so that pressing ESC always takes you back) but even not having keyboard events at all would be fine for a start.

2. Ease of Deployment
The deployment story for Flash is awesome. Send someone a SWF file, he can play it. Yes, for more advanced use cases the file might reference a couple XML files and other asset and might need some parameters to get going but overall that's it.
If you haven't worked in big multi-Agency projects (which would be a good thing :) you might not appreciate this point as important but it really makes sense to have an exchange format for content which one can just "drop" into what is already there.
I don't personally have the perfect solution for SWF like deployment of HTML content but at least I have an idea: Putting HTML files and related assets in a ZIP file and then delivering the ZIP file to the browser which would be smart enough to unzip the file, stick it into an iframe and display everything. I played around with doing all of that with just a little bit of JavaScript. It kinda worked but is nowhere near a fully transparent abstraction.

Tuesday, October 12, 2010

Access to WebSockets on a Sub-Domain though a Proxy iFrame

One of the current design mistakes of Streamie is that the WebSocket server is hosted on the same domain as the main web server. At first this seems like a good idea and the HTTP-upgrade-style implementation of web sockets seem to be made for this approach. Here is why this is still a bad idea:

Scaling web servers is a different task from scaling web socket servers (e.g. because HTTP is inherently stateless while web sockets are inherently stateful and many other reasons). Thus my recommendation is to always host your web socket service on a different domain to enable hosting it on a different infrastructure.

Problem:
While native web sockets make cross domain access easy this is no longer true when you start using all the work arounds (e.g. by using socket.io) like XHR long-polling, multi-part XHR and html-files.
It is important to note that even for clients that do support web sockets often you cannot use them because popular web proxies such as squid do not support web sockets.

Solution:
Access your web socket service through a proxy iframe. This invisible iframe is hosted on the domain of your web socket server and is embedded in your main page. Then access the iframe though the use of document.domain or postMessage. A similar technique (albeit not actually for accessing web sockets is being implemented by #newtwitter. See the source and look for an iframe to api.twitter.com)

Thursday, October 7, 2010

Streamie now runs on Twitter Site Streams

The production environment of Streamie was just switched over to use the Twitter Site Stream API for backend connectivity. (The code for this currently lives in a branch).
With site streams the server only opens a single connection for every 1000 users. Twitter then sends the events of all users over this single connection. The node.js app uses a simple single process message queue to distribute the messages to the actual clients.
If you are running your own streamie server there should be no changes. Site streams are only available to privileged users. If you do not have access, streamie will transparently fall back to user streams. However, the site branch also introduces the usage of a JSON based config file for the server (instead of the ugly command line parameters).

Sunday, October 3, 2010

Joose Blog

I haven't been blogging on Joose here for a while. Check out the official Joose blog for that.

Sunday, September 12, 2010

-O fun || Designing for Hackability

My little side project Streamie was designed to be hackable, as in easy for somebody else to make changes on, from the ground up.
Besides the ability to run Streamie against your own github repository without running your own backend server, there are several other design decisions in the code that are supposed to make the source as approachable as possible.
I'm joining Google next month, so the obvious choice for a client framework would have been to use closure. I decided to go with jQuery instead because everybody already knows that one already.

app.js
When you want to start digging into Streamie, start by reading app.js. In this file you see two big static arrays:
streamPlugins: This is a list of "plugins" which are run on every single tweet and which can transform the tweet in any way. Currently all stream plugins are defined in streamplugins.js. If you want to extend Streamie's tweet view, like e.g. translating all tweets, this is where you want to add your own plugin.
initPlugins: This is a list of plugins which are run as soon as the connection to the backend system has been established and Streamie is basically "running".

The REST API
Twitter exposes an extensive REST API to interact with it's service. Using these APIs requires doing oAuth. Doing anything with oAuth sucks. The good thing is that Streamie provides a proxy for the REST API that does all the oAuth stuff for you. This means you can use the API just as it is documented. E.G. this is how you favorite something in Streamie. Because the main channel from Twitter to Streamie is the user stream API, you don't really have to worry about rate limits.

Tweets, Direct Messages, Retweets
Coming from the Twitter API Tweets, Direct Messages, Retweets look quite different. Streamie normalizes them to be all specialized Tweets. This makes working with them quite easy. Streamie is using the "Stream" metaphor extensively. That is why your direct messages appear in the main stream. The buttons in the navigation only activate CSS classes which filter out certain tweets.

Start up time
Streamie's startup time can be quite long (as in a couple seconds). This is mostly due to the time it takes to load initial data from twitter. (See here if you are curious how that works). In development you can append ?cache to the URL and these API calls will use sessionStorage caching which makes loading almost instant.

Learnings
Overall Streamie's software design is quite simplistic. Right now it feels like there might be a need to abstract some stuff a little more. There will likely be classes for "status inputs", custom events and maybe even tweets :)
Do you have questions or suggestions about how Streamie works or is supposed to work?

Another week in Streamie

Just a quick recap what we did in Streamie this week:
  • Your own tweets now show up with opposite orientation (thanks to @helgri)
  • You can now upload images to be included in statuses
  • We now display your direct messages and you can write them using the standard "d username" twitter syntax
  • We use fancy utf8 characters as icons. This didn't really work on Windows. @helgri added images as a fallback.
  • @joemccann made the "run your own streamie" experience a little smoother
  • If a proxy gets in the way of web sockets we now fall back to more conservative transport methods automatically. If Streamie didn't work for you before, maybe it does now.
Last but not least, @AliceLikesBob provided a patch to integrate chrome notifications into Streamie. This will be integrated shortly.

Next to the direct contributions, people continue to have fun with their own versions of streamie. There are now 27 forks on Github.

Sunday, September 5, 2010

Streaming along

In it's second week of development Streamie is starting to gain quite a few convenience features beyong the simple one like "Yes, we can actually post a status".

The new features are
  • Previews of image links
  • Click on "in reply to" shows ALL tweets of that conversation. This is awesome. Try it!
  • Auto completion of twitter names
  • A page for new users that describes what streamie is
  • Settings
  • ... and many, many small improvements
Hackability++
Streamie is aiming to be the most customizable and developer friendly Twitter client in the world (See how you can make your own version). It is starting to be an active open source project. Streamie now includes contributions from these awesome people: @helgri, @antimatter15, @mathias. Overall there are now 17 forks on github.

Wednesday, September 1, 2010

The first week of Streamie.org


Streamie.org is now one week old. It is already quite usable as a Twitter client and I personally dogfood it 100%. People also seem to like it.

People are also starting to make their own versions of the client (Learn how to do it here). E.g., we will soon get a version of Streamie that actually works well on Windows. The current version uses UTF8 characters which are not included in Windows-Default-Arial.

So what's next
  • Maybe a nice homepage
  • Nicer oAuth dialogs
  • Maybe a command line
  • jQuery style filtering of tweets e.g. $(".status.retweet contains('@phiggins')").hide()
  • Options
  • Direct Messages
If you want to do fun things with Streamie, I suggest you start in app.js and start making your own streamPlugin. E.G. you could color all tweets red that include the word Dojo.
Because Streamie was supposed to be hackable I did actually write source level docs. Let me know what extra info you need.

Saturday, August 28, 2010

The Future is here. I just forked a running website: Streamie.org

I recently started working on a Twitter client. This seems like a really stupid idea because there are so many already. My main motivation was the stale development of the Tweetie for Mac client and that I wanted something really, really hackable. Ideally with the ability to do JavaScript plugins.

There comes Streamie.org
  • A Twitter client with a node.js backend that talks to twitter's brand new user stream API
  • This means updates appear in true realtime in your timeline. No polling involved.
  • When you scroll down, streamie maintains your scroll state as new tweets appear. I can't use a Twitter client without this feature
  • It does some more stuff, but is far from finished.

Now be warned. Streamie is ultra alpha. It will probably crash the moment this blog post appears and it might just post goatse pictures to twitter on your behalf.

If you are not scared, read on:

Ultra Hackable

Now on to the title of this post:
Streamie is extremely hackable. You should never really need to do anything with the backend system (unless you want to fix bugs). The frontend has a access to the full Twitter REST API, so all regular features for a Twitter client should be implementable from the frontend.

Now if you want to build your own version of streamie, there are nine easy steps:
  1. Fork Streamie on Github
  2. git branch gh-pages
  3. git checkout gh-pages
  4. git pull origin gh-pages
  5. git submodule init; git submodule update
  6. edit something in ./public
  7. git commit
  8. git push origin gh-pages
  9. Wait a little on the first time
Now go to YourGithubUserName.streamie.org and visit your new Twitter client.

This is my personal branch.
I love this idea. Maybe it is really stupid and people will make evil clients abusing Streamie's oauth secrets, but maybe this will work and we'll build a great client together :)

This might be a good way to learn all the new hot stuff built into Streamie:

Monday, July 5, 2010

The Evolution of JavaScript

OK, maybe the title is a little too overblown. All I have is two worldles made from the summaries of the submissions to our Call for Speakers at JSConf.eu. You can see node.js and HTML5 coming out of nowhere. Also in 2010 apparently more people spell JavaScript "Javascript" :(

Submissions 2009



Submissions 2010

Monday, June 14, 2010

Crockfordize the Web

Saturday morning I decided to make a multiplayer game with node.js. It turned out to be really easy.
Mind you, this is ultra-alpha software that most likely simply will not work. Also it is hosted on the awesome heroku node.js beta setup which means you might get hit by the pretty narrow request limits.
Anyway, I think it is fun already, so here we go.
Just click here and start shooting at this page. If somebody else is here as well, you should see them shooting as well. You can also take the above link and execute it on any page as a bookmarklet.
To make this more fun, here are some targets:


Wednesday, April 21, 2010

Server Side JavaScript is very real

At the super awesome JSConf 2010 in Washington DC last weekend Ryan Dahl, the brilliant creator of the server side JavaScript platform node.js, that will eventually rule the world, made a remark on the "CommonJS panel discussion" that some people thought was not completely correct: "Server side JS is pretty much only used for toy demos" (quoting from blurry memory). I think he was actually trying to make a different point, but people still asked me to write about my own experience.

I must be one of the lucky few who is actually paid for making large scale web apps based on server side JavaScript (yes you are allowed to write me emails how envious you are).

Our last project was mirapodo. They are a very-real, zappos-like shoe store startup in Germany backed by the 2nd largest ecommerce retailer in the world. I can't really go into numbers but it is serving millions of page impressions per short time frame :)

The system runs on Rhino, which was pretty much the only real option to do SSJS when we started last summer. I don't want to go too deep into the technical details of the backend systems. All I can say is that we had a blast doing it and that we are going to do it again (This time for Gucci).


Since the site is currently German-only I made some green highlights on were it makes sense to click on the site to see it in action. Also make sure to scroll down with endless scrolling, then pick a product and then go back via the back button. The HTML from the endless scrolling is stored in HTML5 local storage and then completely restored without a single AJAX request :)

We try to share as much code between client and server as possible. This is were server side JS for websites really shines. For once we are dogfooding Joose on both sides and probably the coolest example is this Facebook connect app: It is using the official Facebook API for the client side on the server side. (Yes that is why it is so slow :)

BTW: Thanks to SinnerSchrader for letting me do this.

Tuesday, March 30, 2010

Conferences 2010

Things will get a little busier in the coming weeks with several conferences coming up:

JSConf.us
Dunno whether I'll do a talk on track B yet but nevertheless this will surely be the best tech conference ever.

I'll try to do a presentation on why it makes sense to make websites load fast from a business point of view.
Its really cool to be at the awesome SWDC again. I'll talk about why node.js is the best thing since sliced bread.

I think news is out that we'll do it again. More info at JSConf.us.

Wednesday, March 24, 2010

Joose has a new home

Nickolay created a great new home for Joose in preparation of the release of Joose 3.0

Check out joose.it