Thursday, January 12, 2012

On a personal note

Me and my wife Svenja moved to San Francisco last Tuesday. Yaiih :)
So far we've been having a ton of fun and really enjoyed the weather compared to cold rainy Germany.
Svenja has been blogging about her experience coming to San Francisco. She called the blog cramforceswife.tumblr.com after my Twitter name cramforce. Must be true love (more likely she is being ironic :)

Thursday, December 15, 2011

Experimental support for Common JS and AMD/require.js modules in Closure Compiler

I recently stumbled over a thread on the jQuery mailing list about how to modularize jQuery which keeps getting bigger and bigger with every version with not everybody using every feature. Some argued to change jQuery to support "dead code elimination" via Google Closure Compiler's advanced optimizations, which would eliminate unused code from people's projects; others wanted to use AMD/require.js modules instead, which enables only loading required dependencies. Having just done a little project on closure compiler at work, I figured it might be possible to support both of those ideas equally. And so I got coding…

How it works

With my change closure compiler (CC) gets experimental 1st class support for both Common JS and AMD modules. This means that CC knows about these types of modules and performs special transformations and optimizations for them. The high level goals are:
  • Concatenate all modules into a single large file.
  • Automatically order modules so that dependencies are fulfilled.
  • Make it really easy for CC to apply its built in optimizations.

Step 1: Transform AMD to Common JS modules

Add --transform_amd_modules to the command line options of CC to transform AMD modules to Common JS modules. In this first steps basically
define(['foo'], function(foo) { return {
  test: function() {}
}});
gets transformed to:
var foo = require('foo');
module.exports = { test: function() {} }
From now on we don't have to worry about the peculiarities of asynchronous AMD anymore. This step by itself, might be useful to some people. E.g. if you want to use AMD code in Node.js directly.

Step 2: Process Common JS modules

Add --process_common_js_modules to the command line options of CC to enable specific processing of Common JS modules. Most Common JS implementations (like e.g. Node.js) implement it by wrapping all code of a module in a closure like this:
(function(require, exports, modules) { /* your module code */ })(…)
The problem with that is that the module pattern is really hard to optimize for closure compiler because with function calls and scopes involved, everything becomes really dynamic and hard to statically understand. This is why I implemented a transformation for Common JS modules which allows them to be safely concatenated into a single JS file without the use of closures. This works by renaming all global symbols in a module so that they never conflict with a different module. The following Common JS module named "example/baz":
var foo = require('foo');
var bar = require('bar');
exports = { test: 1 };
becomes:
var foo$$example$baz = module$foo;
var bar$$example$baz = module$bar;
module$example$baz = { test: 1 };
Notice how exports just becomes module$example$baz while require('foo') gets turned into module$foo. As you see both exports (and by proxy module.exports) as well as require get converted into direct references to the specific module. All global variables and function names get suffixed with the module name, so that they can no longer conflict with any other module. Note that while these sources seem really verbose, closure compiler will, of course, make all variable names really short later in the compilation process.

Step 3: Managing dependencies

Add --common_js_entry_module=foo/bar.js to your command line options to specify your "base" or "main" module. Going from this, the system will figure out the dependencies and only include those in the final output. Also everything will be in the right order.
find ./dojo -name '*.js' | xargs java -jar ../../build/compiler.jar --formatting=PRETTY_PRINT --process_common_js_modules --transform_amd_modules --common_js_entry_module=dojo/behavior.js
This command shows how you can just use find to pipe all your modules into CC and then have it figure out what you need based on the common_js_entry_module.

Caveats

  • I did this just for fun. However, I'm not actually using AMD or Common JS myself right now, so this isn't battle hardened (but it does compile Dojo). Please help me with feedback and patches.
  • There will be many bugs.
  • It does not support Require.js plugins because those can usually not be executed statically (that is their whole point).

How to use it

Download Closure Compiler and go.

On Performance

I'd argue that if you need to load some JS, doing it in a single requests usually always wins. Having *ALL* your JS in one file is, however, usually not a good idea. You want to incrementally load stuff. How to do that within the framework of what I described above is left as an exercise to the reader.

Comment here

Friday, November 25, 2011

ADVANCED_OPTIMIZATIONS in Google Closure Compiler

Google Closure Compiler supports a compilation mode called "ADVANCED_OPTIMIZATIONS". There are official docs but I wanted to highlight a couple interesting aspects of what activating this mode means in practice. These features set closure compiler apart from other minifiers such as UglifyJS and can potentially help to substantially decrease code size. They do, however, come as trade offs that limit your flexibility with using the JavaScript syntax. It is a common myth that advanced optimizations require using JSDoc types (which are then checked by closure compiler). This is not true.

Property renaming

Closure compiler can rename properties.
foo.longName
becomes
foo.a
Of course, this will break, once you call foo['longName'], so be careful :)

Property collapsing

Closure compiler can collapse property chains (which are often used to represent namespaces in JS)
foo.bar.baz
becomes
foo$bar$baz (and will eventually be minified with standard variable renaming.
There are various exit conditions when this optimization cannot be applied. Never assign to a global object twice. Don't alias objects in the property chain as in: a = {b: 0}; c = a; c.b = 5;
This optimization reduces code size and can increase execution speed because fewer lookups have to be performed.

The catch

Don't use the module pattern. Code, that includes the module pattern usually will not minify well with closure compiler. Yeah, sorry about that :) This is not because the closure compiler engineers do not like the module pattern (and similar constructs) but rather because invariants that have to exist for the optimizations (and others) described above, can not be (easily) shown for code with very complex runtime semantics.
We recently had an awesome conversation of G+ about how to fix this. The idea is to transform special cases of idiomatic JavaScript (in this case using AMD) to a form that can be efficiently compiled by closure compiler. Lets do this!

Comments

Thursday, October 6, 2011

JSConf EU – The How-To Guide

Last weekend was the third JSConf EU. Again it was both an extremely challenging and rewarding experience as an organizer. We are honored to welcome such amazing attendees and speakers to create the ultimate nerd-heaven weekend. Thanks for coming from Jan, Holger and me.

People seem to like our little conference. These are a couple paragraphs about how we put it on. The world still needs more JSConfs. Come to one of ours and then talk to us how to do one at your place of the world.

Done for love, not profit
We spend every dime that we take it on the conference and don’t aim for profit. Often sponsors are signed at the last minute when all vendors are already paid for so it is actually hard to spend more money. In this case we’ll just increase the party and food budget.
Not going for profit makes a whole different conference experience possible. While conferences usually optimize for creating the best possible value for the companies that pay for attendees’ tickets or the sponsors, we can make seemingly irrational decisions like having the best espresso maker in town at our venue that make a big difference for our attendees.

Parties
We always do 3 parties: One before the conference, a big one after the first day (because everyone will have time and be in town) and another one after the conference.
As we are in Berlin we try to bring the original Berlin party experience to our attendees. That means torn down venues in abandoned homes and lots of Techno.
Drinks are always free for all which certainly helps in making the parties great.
We also invite the local Berlin developer scene and friends to the parties. We want to show the people that travelled here how awesome this community is and we want to give the community a chance to meet all the great people who come. This usually works out great.

Speakers dinner and other amenities
Can’t have one because there is a big party for everyone :)
10-20% of our overall attendees are actually speakers. Another large percentage of attendees have spoken at previous installations of the conference. This is another reason why e.g. having a special area on the venue that is exclusive for speakers does not make sense. In conjunction with the No Stage rule below, the ensures that speakers and attendees freely mix and speakers don't camp out hiding. And attendees don't get to feel inferior.

Stage
We do not have a stage. We feel that part of what makes our conference special is that speakers and attendees are on one level and not having a stage to speak on emphasizes this point.

Speaker travel and hotel
We pay for our speaker’s travel. On the other hand, we are happy if the speaker’s employer offers to pay for travel. As we are not-for-profit, we immediately turn around and use the budget to make something else more awesome. We also pay for their hotel, we add two nights around the conference at least (for all, not just international speakers) and we make it a fancy hotel, take care of the wifi and do everything else to make sure speakers are treated like rock stars. We don't quite pick them up from the airport individually, but we are thinking about it. If you treat speakers nicely, they are more likely to bring their A-game talks.

Technical depths of talks
We encourage our speakers to aim their talks at an audience that is made up of experts in their particular field. Thus there won't likely be any “Introduction to X” talks. If one does not happen to be an expert in a given field, it is very likely that one will reach a point in a talk where one does not understand everything. At this point, if you are not a fan of mind blowing brain massages, JSConf is not for you. On the other hand, this does actually enable the conference to push attendees into new levels of understanding and interest. One can always read up on details after the conference and maybe eventually become an expert.

Talk length
Most of our talks are 30 minutes long. The reason is that we want to invite the people who actually created. Not all of them are necessarily experienced public speakers. Doing a 30 minute talk is significantly easier than longer formats which helps keep the quality very high. And if you can't convey your awesome topic in 30 minutes, the attendees are likely to get bored.

Timing
We do the conference on the weekend. While this might not be ideal for families, we make this tradeoff because we are looking for attendees, which are willing to sacrifice a weekend to come. This is also the reason why we start our ticket sales on Sundays.

Size
We did 350 people this year. My personal experience was that this was already a little too big, as such a number of people requires executing everything with great precision which is super unlikely for amateurs like us (Professionals can’t do this either, but they don’t care because lots of attendees means lots of money).
250 people seems like a great number for a conference, smaller doesn’t hurt.

Round Tables
Round tables at a conference make it look more like a wedding and they don't force everyone facing the same direction. Also they encourage conversations, which is our ultimate goal. See also: Building Serendipity

Power under every table
We are geeks and we love our laptops so we make sure we can charge them whenever we want. Next year we should also have power in the lounges as well.

Food
When a caterer says they can provide you with great food, we assume they are lying. Collecting references and testing the food is a must. It is important to have multiple food lines to make sure everybody gets their lunch quick - even if not everyone undstands this within milliseconds. In Berlin, make sure they stock up on Club Mate, and beer.

Venue
The first year we did JSConf EU we got lucky by finding a venue that was perfect for our size, yet very much provided a unique Berlin experience that you can’t easily find anywhere else in the world. Ever since we are looking for raw, industrial venues that don’t even need electricity, let alone an internet connection and then bring in everything to create a truly unique experience. Compare that with a doing a conference in a business hotel that makes you bored even before the first talk has started.
We learned that it is key to have great light artists who can turn every shithole into something very special.

Wifi
Impeccable Wifi is a must. Venues usually don't know/care or even lie about their capabilities. 16Mbit DSL over two routers is not enough for 200 people. Heck, it isn't enough for 20 people. See my article about this topic for suggestions. Do not trust venues unless you tried their wifi with a crowd as big as yours.

Sound
We always hire professional sound engineers for every track. They do things such as turning down the volume when a speaker sneezes which is a small thing but quite valuable when you sit in the audience. Make sure you have headset microphones as they work much better for inexperienced speakers and that you have a monitor box for the speaker.

Video
We rent the most expensive and awesome professional projectors money can buy (even if we then have to save budget elsewhere).
We’ll use DVI next year. Using VGA was a mistake (although not a bad one). Make sure you have VGA-to-DVI-I equipment and a pack of all possible converters on the speaker desk (You’ll also want a Apple power adapter there as usually 95% of speakers use Apple hardware).

Selling tickets
Conferences only sell tickets when they are sold out. This is a fact of life. Should you ever consider to run a conference, count your friends who will definitely buy a ticket. This will be the number of tickets you sell in early bird round one. Make sure your friends buy their tickets in the second sales open and then tweet about it immediately. The rest of the tickets will sell like a charm.
Fortunately we don’t have to pull tricks like this anymore, as we usually sell out all tickets in seconds.
We have three ticket categories: 1. a special rate for attendees of the previous US conference (if you come to both you are truly bad ass) 2. an early bird with a fixed percentage of total tickets, say 20-40% and 3. Regular tickets. There are no student discounts, no press tickets, no community tickets. Here is why: Since all money goes into making the conference, we can make the tickets as cheap as possible. And we feel it is fairer for everybody to pay a low price rather than having some pay more so few can pay less. This means some won't have the funds to attend, but that's how it is. We usually have a number of sponsors running contests around tickets which gives you a chance in.

Sponsors
Sponsors appreciate sponsoring great conferences. Great conferences do not let their sponsors influence their decision-making. The way we make sure this happens, we generally follow this procedure while budgeting: the amount of money taken in from the attendees covers all base costs, venue, speakers, food, everything, but it might mean that we have 3-star food, and only a limited amount of free drinks at the parties. A sponsor then come in and upgrade the food and create a larger drink tab, or pay for another DJ set or something. But they can never interfere with the base conference setup. That said, we do not feel it is bad to have sponsors speak, in fact, we love it. But it is important to be very explicit about the fact, that product advertisements are not acceptable. If in doubt, we check the slides in advance and work with sponsors to make sure they don't miss a chance to totally delight the finest people in our industry.

The unexpected
We always try to do something noone in the audience expects. This might be a musical performance or a talk about nuclear physics.

JSConf.eu opening song - JavaScript Will Listen from Alexander Lang on Vimeo.



Please comment here.

Thursday, June 16, 2011

window.nextTick

Node.js has a special API called process.nextTick which schedules a function to be executed immediately the next time the event loop is idle.
Web applications often try achieve a similar result using setTimeout. The problem is that setTimeout with a zero time parameter does not schedule for immediately but rather for some time a little later. This *can* lead to real performance problems.
I benchmarked a couple ways to do postMessage between workers, iframes and on the current window itself. Turns out doing postMessage on your current window might be a really good alternative to implement process.nextTick in browsers.
Please tell me what I did wrong in my benchmark.

Sunday, June 12, 2011

Triggering de-optimized execution in modern JavaScript engines

Yesterday at NodeCamp.eu I learned from Mr. Aleph how one can easily get a good idea for when V8 has to run a function in un-optimized mode:
curl -s http://v8.googlecode.com/svn/branches/bleeding_edge/src/hydrogen.cc | grep "Bailout("
does the trick.
The result right now is:
owner()->Bailout("bad value context for arguments value");
owner()->Bailout("bad value context for arguments object value");
builder->Bailout("arguments object value in a test context");
void HGraphBuilder::Bailout(const char* reason) {
Bailout("function with illegal redeclaration");
Bailout("Unsupported phi-use");
return Bailout("context-allocated arguments");
return Bailout("EnterWithContextStatement");
return Bailout("ExitContextStatement");
return Bailout("SwitchStatement: too many clauses");
return Bailout("SwitchStatement: non-literal switch label");
return Bailout("ForInStatement");
return Bailout("TryCatchStatement");
return Bailout("TryFinallyStatement");
return Bailout("DebuggerStatement");
return Bailout("SharedFunctionInfoLiteral");
return Bailout("reference to rewritten variable");
return Bailout("reference to uninitialized const variable");
return Bailout("reference to const context slot");
return Bailout("reference to a variable which requires dynamic lookup");
return Bailout("Object literal with complex property");
if (!Smi::IsValid(i)) return Bailout("Non-smi key in array literal");
return Bailout("unsupported const compound assignment");
return Bailout("compound assignment to lookup slot");
return Bailout("invalid lhs in compound assignment");
return Bailout("non-initializer assignment to const");
return Bailout("assignment to const context slot");
if (proxy->IsArguments()) return Bailout("assignment to arguments");
return Bailout("assignment to LOOKUP or const CONTEXT variable");
return Bailout("invalid left-hand side in assignment");
Bailout("arguments access in inlined function");
Bailout("Function.prototype.apply optimization in inlined function");
return Bailout("call to a JavaScript runtime function");
Bailout("delete with global variable");
Bailout("delete with non-global variable");
return Bailout("invalid lhs in count operation");
return Bailout("unsupported count operation with const");
return Bailout("lookup variable in count operation");
return Bailout("Unsupported non-primitive compare");
return Bailout("unsupported declaration");
return Bailout("inlined runtime function: IsNonNegativeSmi");
return Bailout(
return Bailout("inlined runtime function: ClassOf");
return Bailout("inlined runtime function: SetValueOf");
return Bailout("inlined runtime function: RandomHeapNumber");
return Bailout("inlined runtime function: GetFromCache");
return Bailout("inlined runtime function: SwapElements");
return Bailout("inlined runtime function: MathSqrt");
return Bailout("inlined runtime function: IsRegExpEquivalent");
return Bailout("inlined runtime function: FastAsciiArrayJoin");
I was also pleased to learn that my assumption that modern JS engines de-optimize when they see use of "arguments" is no longer true. E.g. uses like fn.apply(this, arguments) which is often used in AOP style function wrapping would be able to run in optimized code. I benchmarked various uses of arguments in this JSPerf testcase. The idea is to have a loop in each test which should be easily optimized but which would take so long to run that the actual usage of "arguments" in the first statement should not influence the benchmark itself. Suggestions how to make this better, appreciated :)

Does anyone know of similar ways to find out about which elements of JS will trigger running in non-optimized mode for other (open source) engines?

Thursday, June 2, 2011

On a personal note

Warning: If you do not know my mom you can quit reading now.

Just wanted to say how proud I am of my mom to quit her job after like 40 years of working, going back to college, finishing that in no time and now open her own practice as a psychologist: Psychologische Beratung Elmshorn