Category: Technology

This is a part of a report I wrote last month; the full report is available here.

Mike Belshe – [email protected] – 04/08/10

When choosing an Internet connection, it is intuitive to focus on network bandwidth – “Speeds up to 5Mbps!”  Bandwidth is important, but it isn’t the only factor in performance.  And given the way that HTTP uses short, bursty connections, it turns out that the round-trip-times dominate performance more than bandwidth does.

Of course, bandwidth speeds are very important for video, music, and other large content downloads.  These types of content, unlike web pages, are able to utilize the network because they are large, whereas web pages are comprised of many short connections.  This report is only concerned with the effect of bandwidth and RTT on the loading of web pages.

Two Factors in Web-page Downloads:  Bandwidth and Round-Trip Time

If we make an analogy between plumbing and the Internet, we can consider the bandwidth of the Internet to be like the diameter of the water pipe.  A larger pipe carries a larger volume of water, and hence you can deliver more water between two points.

At the same time, no matter how big your pipe is, if the pipe is empty, and you want to get water from one point to the other, it takes time for water to travel through the pipe.  In Internet parlance, the time it takes for water to travel from one end of the pipe to the other and back again is called the round trip time, or RTT.

While it is easy to install a bigger pipe between two points to add more bandwidth, minimizing the round trip time is difficult because physics gets in our way.  If your server is located in London, for example, and you are in San Francisco, the time it takes to send a message is gated at least by the speed of light (even light would take ~30ms to travel that distance, or 60ms for a full round-trip).  On the internet today, round trip times of 150ms to 250ms is common.  And no matter how big your pipe is, the RTT will take this long.

Test #1:  Vary the Bandwidth

The first test we ran was to see the effect on web-page download times when you change the bandwidth of the network.  In this test, we’ve chosen fixed values for packet loss (0%) and round-trip-time (60ms) so that we can isolate the variable under test, bandwidth.  In this test we’re downloading a simulation of 25 of the web’s most popular web pages.

Here are the raw results, using HTTP to download the web page:

Bandwidth

Page Load Time via HTTP

Figure 1: This graph shows the decrease in latency (page load time) as the bandwidth goes up.  As you can see, there are diminishing returns as the bandwidth gets higher.   According to the Akamai Internet Report Q2’09, the average bandwidth in the US is only 3.9Mbps.

Figure 2: Here we see the incremental percentage decrease in latency as we add more bandwidth.  An increase from 5Mbps to 10Mbps amounts to a 5% improvement in Page Load Times.

Figure 3: This graph shows the effective bandwidth of the web page download as the raw bandwidth is increased.  At 1Mbps, web pages can be downloaded at ~69% of the total bandwidth capacity.  At 10Mbps, however, web pages can only be downloaded at ~16% of the total capacity; and we’re tapering off.

Test #2:  Vary the RTT

For this test, we fix the bandwidth at 5Mbps, and vary the RTT from 0ms to 240ms.  Keep in mind that the worldwide average RTT to Google is over 100ms today.  In the US, RTTs are lower, but 60-70ms is still common.  

RTT

Page Load Time via HTTP

Figure 4:  In this graph, we see the obvious effect that Page Load Time decreases as the RTT decreases. Continued reductions in RTT always helps, even when RTT is low.

Figure 5: This graph shows the % reduction in PLT for each 20 milliseconds of reduced RTT.  Unlike improvements to bandwidth (figure 2), where benefits diminish, reducing the RTT always helps the overall PLT.  In these tests, with bandwidth fixed at 5Mbps, each 20ms yielded 7-15% reduction in PLT.

Figure 6: This chart shows the increased overall effective bandwidth as RTT is reduced.  With high RTT, bandwidth of a page load was as low as 550Kbps, which is a little more than 10% of the 5Mbps theoretical throughput.  With low RTT, web page downloads still only achieve ~54% of the total bandwidth available.  This is due to other factors of how web pages are loaded.

Conclusions

As you can see from the data above,  if users double their bandwidth without reducing their RTT significantly, the effect on Web Browsing will be a minimal improvement.  However, decreasing RTT, regardless of current bandwidth always helps make web browsing faster.  To speed up the Internet at large, we should look for more ways to bring down RTT.  What if we could reduce cross-atlantic RTTs from 150ms to 100ms?  This would have a larger effect on the speed of the internet than increasing a user’s bandwidth from 3.9Mbps to 10Mbps or even 1Gbps.

This is great news.  All data communications should be encrypted.  You never know who is sitting next to you at the cafe.

http://googleblog.blogspot.com/2010/05/search-more-securely-with-encrypted.html

I use Visual Studio 2005 for my Chromium builds.  Because Chromium is a cross-platform product (Windows, Mac, and Linux), we’ve built tools (robots) which can automatically build our code changes on the other platforms before we ever commit the change.  When a pending change fails on the robots for other platforms, it creates a headache for me because I have to rework the patch and retest.

Unfortunately, on Windows we use warning level 3, which treats Visual Studio warning 4018 as a problem but not Visual Studio warning 4389.  4018 treats signed and unsigned comparisons for greater-than or less-than as warnings, but not equality comparisons.  4389 treats signed and unsigned equality comparisons as warnings.  Why Microsoft split these into two different warnings, I don’t know.  They should be part of a single warning, in my opinion.

g++, which we use on Linux and Mac builds, does not differentiate these signed/unsigned comparisons in the same way.  And this is annoying, because it means that a signed/unsigned equality comparison will seem to work on my Windows machine, and then fail on Linux and Mac (“warning: comparison between signed and unsigned integer expressions”!).

If you have a similar problem, I recommend using Visual Studio’s option for “/we4389”, which will include warning 4389.  And alas – your builds on all platforms will treat unsigned/signed comparisons the same way.  Phew!

Jobs puts forth some pretty good arguments here for why the iPhone and iPad don’t support Adobe’s Flash.  From the driver’s seat (iPhone is at the top-of-the mobile device heap), you can see why Apple doesn’t want Flash holding it back.

When browser upgrades arrive, they usually contain good improvements in terms of speed, security, reliability, and features.  For website developers, new browsers usually offer better capabilities for building websites.  Unfortunately, there is a lag-time between when browsers are released with new features and when users upgrade sufficiently that developers can use them.

Upgrades are also important from a security perspective.  When users run older browsers they are exposed to security risks which have been previously discovered and potentially fixed.  In the case of Internet Explorer, this problem got so bad, that various governments, and even Microsoft advised against users’ continued use of Microsoft Internet Explorer 6.

So how long does it take users to shift to a new browser?

The data from StatsCounter shows this very clearly.  Chrome upgraded virtually all of its users from version 4.0 within a month, starting in Feb 2010.  Firefox also upgraded version 3.6 at about that time.  The upgrade is progressing quickly, and now, 3 months later, approximately two-thirds of Firefox users have upgraded.   Internet Explorer 8, by contrast, was released over a year ago, and so far less than half of IE users have upgraded.

This week some researchers from Microsoft published a paper about a project called JSMeter they have created for measuring JavaScript performance.  In the paper, researchers Ben Livshits and Ben Zorn want to examine the effectiveness of current Javascript benchmarks and how accurately they reflect “real world applications”.  This is a great topic, commonly debated, and one which we know we should do better at.

I like their intentions and their idea, except for one fundamental flaw.

To understand, let’s compare their research to how a mechanic might benchmark the performance of two cars.

In this case, we’ve got a Chevy Passenger Van and a Porsche 911.¹ To measure the performance of the cars fairly, we take them to the track.  We recognize that the track is not indicative of real world driving, but it does give us a place to compare each car under the same conditions.

Normally, you’d probably want to drive both cars around the track, right?  Sure.  But these researchers decided to only drive the Van around the track.   Despite the obvious fact that the performance characteristics of the Van have little bearing on the performance characteristics of the Porsche, they then used the performance of the Van to make claims about how Porsche should be tuned and the track should be improved to be more like real driving conditions.  This claim is absurd.

In their last test, the researchers decided to drive both the Porsche and the Van around the track.  But in this test, they elected to have an elephant sit on top each car as it went around the track.  Rather than observing that the Porsche carrying an elephant is still faster than a Van carrying an elephant, they document the fact that the Porsche with an elephant is 2x slower than the Porsche without an elephant, while a Van with an elephant is only 30% slower than a Van without an elephant.

Wow.  Read the report for yourself, this is exactly what they did.

Now, don’t get me wrong – I’m not defending the existing benchmarks in any way.  We definitely need more and better benchmarks.  And their research, when done properly, will likely prove their hypothesis – that the existing benchmarks don’t accurately reflect real world websites.  (I thought we already knew that?) 

1. IE8’s JS engine has been well documented to be orders of magnitude slower than any other JS engine on every single test, so I believe the Passenger Van is a reasonable comparison; if there is a flaw, it is that the Van is too fast for this analogy (it’s not 10x slower than a Porsche), and I should have used a moped.

Note:  These views are mine alone and do not reflect those of my employer.

As part of my SPDY work, I published an informal slide deck about the effect of TCP’s slow start on HTTP performance.

Cliff notes:

1. It has been known that TCP’s slow start adversely effects performance in high-latency, high bandwidth networks for years.
2. Increasing cwnd (reducing slow start) has been slow through standards due to concerns about internet collapse.
3. But web servers and browsers have already worked around TCP’s slow start by pummeling the net with excessive connections – effectively making slow start irrelevant.
4. If slow start has already been worked around, and the internet has not collapsed, it is time to seriously look at changing how slow start works so that we don’t have to open 30 connections in order to have a low-latency transaction.

Feedback is welcome!

I installed FreeBSD for the first time in a long time today.  I had to install it twice.  That sounds really bad, but it was for a surprising reason.

The install was pretty simple from the DVD; it kept asking me questions, and I answered them quickly, and after about 5 minutes, the entire process was complete.  Because the process was so fast, that I thought I had done something wrong (perhaps I had been bit by the cluky ANSI installation interface), so I installed again.  But again, it was a quick process and only took 5 minutes and then it returned to the main install screen.  Puzzling.

After a minute of thinking, I decided to try to boot.  And it worked!

So the entire from-scratch install process only took 5 minutes!  I certainly didn’t expect any sluggish Microsoft-like install times, but 5 minutes was amazing.  I guess they needed a bigger “I’m done” screen so that idiots like me won’t have to install twice.