Saturday, December 6, 2008

Massively parallel crowd-sourced JavaScript app server cloud

Reporting live from the "Why? Because you can!"-department:
Probably due to the fact that I watched way too many episodes of Terminator - The Sarah Connor Chronicles, seeing people talk about Skynet, or because I really wanted to build Google App Engine applications with JavaScript or because I have way too much time on my hand, I decided to build a JavaScript application server that runs within the browsers of the people that visit the site that the app server is delivering. Are you following me? :)

The system works as follows:
  • When a user makes a request to the site
  • ...the request is handled by a Google App Engine application
  • ...that puts the request into a request queue
  • Meanwhile on another computer that runs the JavaScript app server within a Google Gears worker within the browser
  • ...a comet-client maintains a pseudo permanent connection to the App Engine app
  • ...looks into the request queue and starts to process the request
  • ...when it is finished it sends the response to the App Engine app
  • ...which immediately send the request back to the client that originated the request in the first place.
Crowd-sourced cloud computing
  • Ressouce requirements on the server side are reduced because every client can share some of its own ressources with every other user of the system
  • Works with my favorite language JavaScript
  • Enables absolutely seamless integration of client- and server software
  • Every app server is totally untrusted.
  • For security reasons the response of the JavaScript server application must be JSON. It cannot be plain HTML, because every untrusted client could inject evil scripts into the stream
  • Google App Engine unfortunately does provide a good way to implemented comet-style applications with long-polling because they enforce very strict rules with respect to request timeouts
  • The App Engine SDK builtin-web server is not multi-threaded. Thus the application can not be tested (easily) within the SDK but has to be deployd to the production system.
  • My implementation is very-very proof-of-concept do-the-easiest-thing-that-could-work style with no error checking and various evil race conditions with respect to the request queue.
  • There is currently no builtin way to persist data. However, applications may of course use any kind of web-service that is accessible via JavaScript.
  • There is currently no way to read-or-write HTTP headers from the applications
Demo Client
You can try the the demo application (a simple counter here). It will only work if you either started the server yourself (below) or if somebody else is currently running the server application.

Demo Server
The server is running in this iframe:

The app engine application might be disabled due to overuse of quotas. If that is the case, I'm sorry.

If you are interested in come code, check out the implementation of the server.


Roberto Saccon said...

cool ! I experimented myself with this kind of stuff ! Maybe it will be against the terms (as long as the appengine infrastructure is not optimized for microthreads) if it is getting popular to to fill out the 10 sec max request duration !

Nick said...

You might be interested in

Unknown said...

That is indeed interesting. This seems to be a pretty direct implementation of you paper :) For which school is it?

Nick said...

waikato uni, new zealand - was my honors report