Linux Client TCP Stack Slower Than Windows

Conventional wisdom says that Linux has a better TCP stack than Windows.  But with the current latest Linux and the current latest Windows (or even Vista), there is at least one aspect where this is not true.  (My definition of better is simple- which one is fastest)

Over the past year or so, researchers have proposed to adjust TCP’s congestion window from it’s current form (2pkts or ~4KB) up to about 10 packets.  These changes are still being debated, but it looks likely that a change will be ratified.  But even without official ratification, many commercial sites and commercially available load balancing software have already increased initcwnd on their systems in order to reduce latency. 

Back to the matter at hand – when a client makes a connection to a server, there are two variables which dictate how quickly a server can send data to the client.  The first variable is the client’s “receive window”.  The client tells the server, “please don’t exceed X bytes without my acknowledgement”, and this is a fundamental part of how TCP controls information flow.  The second variable is the server’s cwnd, which, as stated previously is generally the bottleneck and is usually initialized to 2.

In the long-ago past,  TCP clients (like web browsers) would specify receive-window buffer sizes manually.  But these days, all modern TCP stacks use dynamic window size adjustments based on measurements from the network, and applications are recommended to leave it alone, since the computer can do it better.  Unfortunately, the defaults on Linux are too low. 

On my systems, with a 1Gbps network, here are the initial window sizes.  Keep in mind your system may vary as each of the TCP stacks does dynamically change the window size based on many factors.

Vista:  64KB
Mac:    64KB
Linux:    6KB

6KB!  Yikes! Well, the argument can be made that there is no need for the Linux client to use a larger initial receive window, since the servers are supposed to abide by RFC2581.  But there really isn’t much downside to using a larger initial receive window,  and we already know that many sites do benefit from a large cwnd already.  The net result is that when the server is legitimately trying to use a larger cwnd, web browsing on Linux will be slower than web browsing on Mac or Windows, which don’t artificially constrain the initial receive window.

Some good news – a patch is in the works to allow users to change the default, but you’ll need to be a TCP whiz and install a kernel change to use it.  I don’t know of any plans to change the default value on Linux yet.  Certainly if the cwnd changes are approved, the default initial receive window must also be changed.  I have yet to find any way to make linux use a larger initial receive window without a kernel change.

Two last notes: 

1) This isn’t theoretical.  It’s very visible in network traces to existing servers on the web that use larger-than-2 cwnd values.  And you don’t hit the stall just once, you hit it for every connection which tries to send more than 6KB of data in the initial burst.

2) As we look to make HTTP more efficient by using fewer connections (SPDY), this limit becomes yet-another-factor which favors protocols that use many connections instead of just one.  TCP implementors lament that browsers open 20-40 concurrent connections routinely as part of making sites load quickly.  But if a connection has an initial window of only 6KB, the use of many connections is the only way to work around the artificially low throttle.

There is always one more configuration setting to tweak.

SSL, Compression, and You

One aspect of SSL which many people are not aware of is that SSL is capable of compressing the entire SSL stream.  The authors of SSL knew that if you’re going to encrypt data, you need to compress it before you encrypt it, since well-encrypted data tends to look pretty random and non-compressible. But even though SSL supports compression, no browsers support it.  Except Chrome 6 & later.

Generally, stream-level compression at the SSL layer is not ideal.  Since SSL doesn’t know what data it is transporting, and it could be transporting data which is already compressed, such as a JPG file, or GZIP content from your web site.  And double-compression is a waste of time.  Because of this, historically, no browsers compressed at the SSL layer – we all felt certain that our good brothers on the server side would solve this problem better, with more optimal compression.

But it turns out we were wrong.  The compression battle has been waging for 15 years now, and it is still not over.  Attendees of the Velocity conference each year lament that more than a third of the web’s compressible content remains uncompressed today.

When we started work on SPDY last year, we investigated what it would take to make SSL fast, and we noticed something odd.  It seemed that the SSL sites we tested (and these were common, Fortune-500 companies) often were not compressing the content from their web servers in SSL mode!  So we asked the Web Metrics team to break-out compression statistics for SSL sites as opposed to unsecure HTTP sites.  Sure enough, they confirmed what we had noticed anecdotally – a whopping 56% of content from secure web servers that could be compressed was sent uncompressed!

Saddened and dismayed, the protocol team at Chromium decided to reverse a decade long trend, and Chrome became the first browser to negotiate compression with SSL servers.  We still recognize that compression at the application (HTTP) layer would be better.  But with less than half of compressible SSL content being compressed, optimizing for the minority seems like the wrong choice.

So how do you know if your browser compresses content over SSL?  It’s not for the faint of heart.  All it takes is your friendly neighborhood packet tracer, and a little knowledge of the SSL protocol.  Both the client and the server must agree to use compression.  So if a server doesn’t want to use it (because it may be smart enough to compress at the application layer already), that is no problem.  But, if your server uses recent OpenSSL libraries, it can.  And you can detect this by looking at the SSL “Client Hello” message.  This is the first message sent from the client after the TCP connection is established.  Here is an example from Chrome, viewed with Wireshark.

sslchromeheader

So, does this help?  Sites that use compression at the application layer don’t need this, and it has no effect (those are in the minority).  For servers that opt out, it has no effect.  But for servers that opt in,   HTTP headers do get compressed a little better, and any uncompressed data stream gets compressed much better.

I did find that https://www.apache.org/ is a showcase example for my cause. Apache.org runs a modern SSL stack and advertises compression support on the server, but they forgot to compress jquery.js.  If you load this site with Chrome, you’ll only download 77,128 bytes of data.  If you use IE8 (firefox will be similar), you’ll download 146,595 bytes of data.  Most of this difference is just plain old compression.

For the 50% of site owners out there that can’t configure their servers to do compression properly, don’t feel too bad – even apache.org can’t get it right!  (Sorry Apache, we still love you, but can we turn mod_gzip on by default now? :-)

No Political Efficiency since 1913?

We currently have 435 legislators in the House of Representatives.  This number has been fixed since 1913.  Question:  Do we need the same number of representatives that we had back then?

On one hand, you could argue that we need more seats in Congress.  After all, there were only 97M Americans in 1913.  Today we have 307M Americans.  Surely more constituents requires a larger congress?

But think about the technology advancements since that time.  In 1913, if you wanted to communicate with your representative, what choices did you have?  He certainly didn’t visit his local district very often – the first commercial flight didn’t even take place until 1914.  Calling your representative was unlikely – there was no long distance from California to Washington at back then, and long distance calls from closer geographies were manual and time consuming.  And of course there was no internet, so real-time communication was impossible.   We did have the one-way megaphones of newspapers and magazines.  And of course, you could write a letter. 

So, in 1913, maybe we needed 435 legislators.  Each had a significant job to do with just communicating, corresponding, traveling, and coordinating between Washington and his local region. 

But today, do we need so many?  With a single email, legislators can reach far more than 225,000 people right from the comfort of his mistress’ bed.  Websites, telephones, television, and email combined certainly make the communication burden almost non-existent compared to 1913.

Obviously, there is more to legislation than just communication with constituents.  But, given the gridlock in Washington, the skyrocketing costs of Washington, and the increased dissatisfaction with the never-ending burden of an increasingly complex set of laws, maybe we should cut that 435 in half.  Any reason why not?  Or is that just the way we roll around here?