At JSConf 2011 Meno Abels and I tried to make it work. All the credit really goes to networking guru and all-things-software Meno. Also thanks to Stephouse.net for awesome work on the connectivity and the access points. In the following paragraphs I will walk through the challenges one faces when it comes to WIFI at conferences. This article will stay quite shallow technically. If I ever have more time I will dig deeper.
0. Basics
Dear conference organizer, this is the part that can be easily fixed: You need to have a person that takes care of the WIFI as a core priority. The WIFI will break and you will need someone who is willing and capable to put on the wetsuit and jump into the shit. E.g. at least one of your access points will be overwhelmed at any given time. Many things can be done to fix this. Somebody will have to do it.
Dear conference organizer, this is the part that can be easily fixed: You need to have a person that takes care of the WIFI as a core priority. The WIFI will break and you will need someone who is willing and capable to put on the wetsuit and jump into the shit. E.g. at least one of your access points will be overwhelmed at any given time. Many things can be done to fix this. Somebody will have to do it.
1. Never trust anyone
Now you booked a venue and they say that they can handle the WIFI for you. Chances are, they are lying. In any case, if you want to go this route, ask for references and call the references. Purespace wanted to handle the Nodeconf traffic over a 2Mbit/s down, 300 kbit/s up DSL connection. That will not work. Chances are you have more than 300kbit/s in TCP ACK packages for your downstream traffic. At nodeconf Stephouse saved the day by coming in on 5 minute notice and installing a microwave link in about 45 minutes (Remember the wetsuit thing from above).
2. Number of attendees and number of devices
Calculate 2.3 devices per attendee.
The primary problem this creates is overwhelmed access points. What can you do to fix it:
- Monitor the number of people attached to each access point.
- Add another access point in zones where more people come together than you expected.
- Use as many access points as possible. Lower the antenna power as much as possible to decrease the range of each AP.
- Never have two APs close to each other on adjacent channels.
- Don't use encryption. This will be painful, but it really relaxes the CPU of your APs.
- People sometimes move in groups and stay attached to an AP while others move in and go onto the AP because it is the closest one. This creates an uneven distribution of people over the APs.
Easy fix: Throw everyone off the AP. The devices will then try to connect to the closest AP again.
3. Unknown Building, Temporary Setup
Conferences are by definition temporary setups. You will have no time to tune your system under real world conditions. But still, you will need to update the configuration throughout the conference to make things work better under real load.
4. Bandwidth
Calculate 100KBit/s per attendee in both directions.
You can live with a little less up-link traffic but don't go with consumer level DSL. In case you cannot get that kind of bandwidth from a single provider, take all you can get from multiple vendors and use the software that Meno built for JSConf.eu to aggregate the links (Warning: Only an option if you have a black belt in networking kung fu.)
5. Bandwidth in the Air
Effective bandwidth per WIFI channel is 20 MBit/s. WIFI has 11 channels. That means that you will never get more that 220 MBit/s in the air, ever. This bandwidth has to be shared by everybody in the room. If you put 5000 people in a relatively small room, then your WIFI will be slow. There is no way around that. It is simply physics. (You may be able to add the 5GHz channels, but we recently experienced problems with some Apple devices.)
6. People
People are the primary problem for conference WIFI. Actually not people but the software running on their computers which they did not disable. This involves bittorrent clients and backup software. One person at JSConf uploaded 9.6 GB in the first 30 minutes of the conference. This means that one person used almost 80% of all bandwidth. Identifying these "powerusers" will get your conference a long way to good WIFI. Usually you will only have an IP address, so your only option is to either block the person entirely or to block the remote IP where the traffic is going. At JSConf we introduced social traffic which links all traffic to Twitter identities. This way we can used Twitter @-messages to ask people to disable rogue services.
7. JSConf learnings
All of the above we learned at previous confs. This is what we learned at JSConf:
With a few 100 people at a conferences there will be a couple stupid persons in the audience. Redirecting all traffic from http://twitter.com to https://twitter.com goes a long way in fixing this problem.
Using an auth service such a social traffic requires white listing of IP addresses. Make sure you highjack all DNS traffic (all on port 53 regardless of DNS server used) so that you are able to control e.g. the IP address you white listed for twitter.com.
Now go back and read point 0. The most important thing is to have somebody who cares and gets their hands dirty when it is most needed.
Posts
28 comments:
So 6. means only people with twitter accounts could use the wifi? Seems a little bit over the top... Didn't you have any complaints because of that?
Nope, we didn't. Might depend on the audience. Could be changed to any other oAuth or Open ID provider instead.
Did you have issues with rogue access points (i.e. people forgetting they had set up their phone as a tethered hotspot) ?
No, we haven't had that problem yet. People don't really leave that on by accident because it uses too much power. It might lead to a vicious circle when your WIFI goes down and more and more people switch to tethering.
In all the of the conferences I've setup, the biggest issues are the conference halls/hotels. They have very explicit exclusions in their contracts that prohibit you from bringing in your own internet, and/or mounting your equipment to serve your attendees WiFi needs. Also, from my own experience, it always seems that the biggest problems are the bandwidth to/from the internet, rather than the WiFi units to the end users (except when the conference center thinks that 1 AP for each 100 people is acceptable).
Getting a FAST pipe is a real delicate art. In most areas, the ISPs don't want to do a huge build-out for a temporary setup. Heck, most will even scoff at running anything more than a consumer-grade DSL or cable connection unless you pay thousands or tens-of-thousands to buy all the equipment. This is even a bigger problem in mid-sized to smaller cities where they just don't have much bandwidth to start with...
@quetwo: Absolutely right. The bandwidth problem can be solved with Sashimi. We used it e.g. to aggregate 2 sync 10MBit/s microwave links with 4 consumer grade DSL connections.
About the venues who don't allow your own WIFI: That is why we never go to "standard" venues.
Having done this before:
- Don't use cheap access points (apple, dlink, netgear, all out) if you have more than 20nodes/AP.
- Have dedicated and redundant DHCP, and the ability to squelch the runaway lease consumer(s) you will have.
- Throttle _everyone_ to a manageable speed, and have dedicated _wire_ for the presentation podium.
- Apple (and really, most vendors)+N is flaky depending on vendor+firmware, so test beforehand with a few different OS/models if possible.
- Don't bother with anything above WEP unless _really_ necessary, and warn people it's not a highly secure system.
- Keep your channels named separately (Foo-{G,B,N}) and on different frequencies; People will generally figure out to pick one they can see, and many clients will "frequency hop"
B/G channels overlap, so you really only have 3 (or 4) non-overlapping channels, use them accordingly, but space them out physically.
Ext Antennas are better then builtins
Be above the crowd, not below.
Hope it helps someone.
All good points, except Foo-{G,B,N}.
Separately named networks is likely to have your APs drop one after another. You have a small problem on one AP. Everybody moves over. Creates a vicious circle and kills your network.
SSID roaming works perfectly. No reason to have more than 1.
Interesting post!
Regarding Bittorrent and backups; what about a firewall blocking all but the most essentials ports (80, 443, maybe a few others) and capping the bandwidth? Something like pfSense would probably solve most of these non-wifi-specific (as far as I can tell) problems. On the other hand, it probably requires more hardware.
A good article, but there's many problems in here.
Channels:
There are not 11 channels available. Channels overlap, and putting users on adjacent channels will cause loss of throughput. In the 2.4Ghz band, channels: 1,6,11,14 are non overlapping.
It's best to arrange channels as far away from each other as possible. For example, you might do a plan like:
1 6 11
11 1 6
If you had six APs in the same room.
Reduce power to as low as possible on each AP, and, only accept connections at the highest possible speed. That will force clients to shift to other APs as they move.
HTTP vs HTTPS:
Transparently redirecting people to https to save them from firesheep is solving the problem far too late in the game. Once one packet is transmitted over HTTP, the user's session open for capture and replay.
AP's in Hallways: Use Mic stands or tripods to hold up the AP and it's antennas. No mounting necessary.
Better yet, just buy some xirrus boxes ;)
@John Adams: I was giving an upper bound for the band width. In reality you are right, one can only do 4 channels. We are using that exact channel layout. The point about low power is in the article.
I personally think the redirect is a pragmatic solution. People who care about real security can use VPN.
Xirrus can do it... They kept up at Interop this year nicely.
are we seriously talking about channel layouts in 2011? Controller based-architectures solve most of the problem (with appropriate policies) without thinking. Now focus on AP specs and density and conferences magically get much easier.
W/r/t the -B, -G, -N spot, my experience is that if you have N/B|G, clients often roam between them poorly and en-masse; By having multiple channels, people become your stochastic mechanism.
Still, very nice article!
Cheers
http://blog.jeffreymcmanus.com/157/learnings-is-a-stupid-stupid-word/
On Encryption: Most conferences will use a pre-shared key. What people don't realize is that given the PSK and observation of the handshake, an attacker can calculate the session key and decrypt all traffic between a user and the access point.
So turning off encryption is really not that bad when you consider that all it did was hog CPU and give people a false sense of security.
there are only 3 non overlapping chanels for 802.11 b and g not 11!
You just need to hire some one who has done the cisco wireless CCNA
This works for a tiny tech conference where someone is dedicating themselves to managing the network full-time. What do you do at a 20,000-30,000 attendee convention?
On solving the bandwidth sharing issue: the monowall firewall distribution has a one-click "share bandwidth evenly" traffic shaper option that is really easy to use and powerful.
It also has a builtin captive portal and can integrate with radius.
I run them on yawarra.com.au devices ($300 per device but they can move around 70mbps easy)
Xirrus rocks we are using them on the Bon Jovi Tour and for every Live event we do. 5 Arrays cover an arena with 18,000 attendees our Live-Fi platform solves every problem noted here. Try 88,000 unique devices at the US Open in 2010 or 25,000 over three day at ACL festival. We do big or small / focused events like Science On line... bring it on and get online with @SignalShare
I immediately send it to my system admin. The down speed in the place I come from is a premium at 8 MBps and up speed would be 5 costing around $100 for a month or so with a cap of 50 GB. And if we have a tech conference with around 300 attendees with atleast 200 wifi devices lets say at the rate of 200 X 100 Kbps it comes to 20000 Kbps , 20 Mbps ... We will have to go for 4 connections.. Just calculating.. Thanks
Thanks for the article and congrats on your success.
A few observations though:
1.) Having personel supporting the network is a must, no matter the equipment used. Good point!
2). Have your specific requirements written down and presented to the organizer. Make sure that the organizer is responsible if the expectations are not met. Make it clear that this super important! (Apple does this for their events and training for example).
And have a plan B prepared!
3). RF planning is the most underrated in setting up the WiFi networks. Having to many AP can be as counterproductive as not having them enough. One has to understand that RF is a shared medium (collision domains). Lovering the AP Tx power is okay but you have to remember that signal at lover data rates travel far. A proper design incorporates that.
I think providing security for your attendees is very important so I would provide them with secure connection. While WPA2 is not crackable by it self it is not a secure solution because WPA2 PSK is given to all atendes and attacker can easily sniff the 4way handshake and than decrypt in real time. One can use 802.1X security but than you will need some extra equipment (AAA server) and very good step by step tutorial for your attendees. Third, the easiest, is using personal personal pre shared keys but this is proprietary and only used with Ruckus and Aerohive products.
Most controller based systems provide with load balancing and band steering functionalities. DOSing your clients is effective but very crude way of doing it.
4.) I do not understand what you mean by effective bandwidth per channel (per channel and not AP as some people understand). The throughput depends on the technology used. HR-DSSS (802.11b) has about 5 Mbps throughput, ERP-OFDM (802.11g) has about 20 Mbps and HT (802.11n) has about 70 Mbps (using 20 Mhz channel width and 2 spacial streams). One has to remember though that it takes two to tango. The lovest denominator will determine the maximum throughput per communication path.
Using airtime fairness is also very important in throughput. If a 802.11n AP supports b/g/n clients (they are active on the network) the speed will be reduced to 802.11b speeds. Airtime fairness corrects that.
There are only 3 non overlapping channels in the 2,4 band (1,6,11 or 1,7,14...). So you get the aggregate throughput of 3 channels not 11. In contrast all channels in 5 Ghz band are non overlapping. In some cases one can use 4 channels (1,5,10,14) but there is overlap between the channels and there will be collision. On highly used networks this setup is debatable. There is also an advantage of using 1,6,11 setup. This setup will support clients (stations) from all over the world. For example FCC (US) allows only channels 1-11. Channel above ch. 11 will not be recognized by an FCC approved device.
There are ways of designing the network so you can provide more bandwidth. Using sector antennas or sectorised array products (such as Xirrus) or smart antennas system like Ruckus wireless beam forming. But this can get pretty complicated and is a reason why a good WLAN engineer is payed good money.
There is always a problem with Apple devices so you should pretest your network as best as you can and optimize accordingly. 5 Ghz is a good way to speed up your network.
Also using as few SSIDs as possible will reduce overhead. If one needs segmentation, dynamic VLANs are preferable.
Just my 5 cents :).
Regards,
Gregor
There are 3 non-overlapping wifi channels, not 11. That's important to know when you lay out the APs. Channel 4 and 5 are 85% overlapping.
I second the recommendation to avoid dlink/netgear/apple kind of stuff and avoid encryption wherever possible.
Another good idea is to set up the 5 GHz band as well. A portion of your users will use that band instead (10-20%?) which means less stress on your 2 GHz equipment.
How much would you say you paid in total to provide wifi to all of your conference users, including the time of your dedicated resource? What did this break down to as a cost per user? Finally, was that cost worth it?
Thanks! Great article!
If you are doing events and at the whim of an event venue's A/V supplier, what can you do to ensure the best setup?
I do events of ~300 with 1.5 connections per person on average. Should I ask for:
- # of AP
- speed of AP
- placement of AP in rooms/hallways
?
@Matt: For connectivity and AP hardware expect something in the order of $3000 to $10000 depending on what you need. We can't and we won't measure the time it took to build this because it includes two complex open source projects and we do it for the love of it.
@Jak: You look for a new venue and follow the instructions in this article.
One trick I learned from the Xirrus reps is to use the same SSID for both 5Ghz and 2.4Ghz, but turn 5Ghz up much louder. There's a lot more channel space in 5Ghz and much less interference from the preponderance of 2.4Ghz gear that always bleeds into the conference space on channel 3 and 8... Clients that run 5Ghz will select the stronger signal, and clients that don't wind up on a less-congested 2.4Ghz.
There aren't eleven channels available.Channels overlap, and putting users on opposite channels will cause loss of throughput.
Post a Comment