Wednesday, February 5, 2014

Since everyone seems to be sharing their good-old days data center war stories, here is my favorite: This was 2008...

Since everyone seems to be sharing their good-old days data center war stories, here is my favorite: This was 2008 when the internet was already pretty mature but online video and especially live video was still somewhat newish. A very creative agency came up with an idea to do a 21 day 24/7 live show driving up excitement for their customer's new thing – of course, they had no idea what it actually meant to produce such a thing, so they eventually reached out to us. While also having no idea what we were doing, we at least had just failed to successfully launch a live streaming internet startup, and so we agreed to pull this thing together in like 4 weeks.

Now, this would probably have been not very exciting, had the creative agency not had one actually very good idea: How to market the live stream? Just book every banner slot in Germany (i.e. the homepage of GMX) and show the live stream with no explanation. The banner had the what is likely a record in internet history: A click through rate of over 30% on the first day. This, of course, is mostly because of the novelty of live internet video in those days, but in the end it meant that we had to serve live video to what was a large percentage of people using the internet in Germany.

Actual serving was done by limelight networks (they got large by serving video for early YouTube). I attached a graph I still had around which shows 9 Gbs, which would have been about 20-30k concurrent streams. I don't remember what our actual peaks were.

The rest of the pictures show our on-site infrastructure. Everyone in advertising being cool, this was in a random house in some hip and gentrifying part of Hamburg, not in a TV studio. We had to get actual internet connectivity there. The pictures show our directed radio link and the even more horrible consumer DSL uplink (we used those redundantly).

Actual video production was still analog, thus the pictures show the encoders attached to 13 servers (2 streams, 3 bit rates, 2 x hot redundant (1 streaming via directed radio and one via DSL), 1 spare) which ran the On2 VP6 encoder on Windows (shiver).

The flash players we developed would automatically pick on of the redundant streams and fail over to the other one if there was a failure or move to a different bit rate. Power was not in any way redundant – the servers had a UPS, so they would not die hard upon power failure, but I think the cameras and video editing equipment was just plugged into the wall – but power did not fail :) The on2 encoders sometimes crashed but this wasn't a problem with the redundant streams and I think the direct radio link failed once, but overall this was a very successful project which shows that you can do everything if you are just sufficiently na├»ve.