When you turn on IPv6 in your operating system, the web is going to get slower for you. There are several reasons for this, but today I’m talking about DNS. Every DNS lookup is 2-3x slower with IPv6.
What is the Problem?
The problem is that the current implementations of DNS will do both an IPv4 and an IPv6 lookup in serial rather than parallel. This is operating as-per the specification.
We can see this on Windows:
TIME EVENT
0 DNS Request A www.amazon.com
39 DNS Response www.amazon.com
39 DNS Request AAAA www.amazon.com
79 DNS Response www.amazon.com
<the browser cannot continue until here>
The “A†request there was the IPv4 lookup, and it took 39ms. The “AAAA†request is the IPv6 lookup, and it took 40ms. So, prior to turning IPv6 on, your DNS resolution finished in 39ms. Thanks to your IPv6 address, it will now take 79ms, even if the server does not support IPv6! Amazon does not advertise an IPv6 result, so this is purely wasted time.
Now you might think that 40ms doesn’t seem too bad, right? But remember that this happens for every host you lookup. And of course, Amazon’s webpage uses many sub-domain hosts. In the web page above, I saw more of these shenanigans, like this:
TIME EVENT
0 DNS Request A g-ecx.images-amazon.com
43 DNS Response g-ecx.images-amazon.com
43 DNS Request AAAA g-ecx.images-amazon.com
287 DNS Response g-ecx.images-amazon.com
Ouch – that extra request cost us 244ms.
But there’s more. In this trace we also had a lookup for the OCSP server (an Amazon’s behalf, for SSL):
TIME EVENT
0 DNS Request A ocsp.versign.com
116 DNS Response ocsp.versign.com
116 DNS Request AAAA ocsp.versign.com
203 DNS Response ocsp.versign.com
Ouch – another 87ms down the drain.
The average website spans 8 domains. A few milliseconds here, and a few milliseconds there, and pretty soon we’re talking about seconds.
The point is that DNS performance is key to web performance! And in these 3 examples, we’ve slowed down DNS by 102%, 567%, and 75% respectively. I’m not picking out isolated cases. Try it yourself, this is “normal†with IPv6.
What About Linux?
Basically all of the operating systems do the same thing. The common API for doing these lookups is getaddrinfo(), and it is used by all major browsers. It does both the IPv4 and IPv6 lookups, sorts the results, and returns them to the application.
So on Linux, the behavior ends up being like this:
TIME EVENT
0 DNS Request AAAA www.calpoly.edu
75 DNS Response www.calpoly.edu
75 DNS Request A www.calpoly.edu
93 DNS Response www.calpoly.edu
In this particular case, we only wasted 75ms, when the actual request would have completed in 18ms (416% slower).
But It’s Even Worse
I wish I could say that DNS latencies were just twice as slow. But it’s actually worse than that. Because IPv6 is not commonly used, the results of IPv6 lookups are not heavily cached at the DNS servers like IPv4 addresses are. This means that it is more likely that an IPv6 lookup will need to jump through multiple DNS hops to complete a resolution.
As a result, it’s not just that we’re doing two lookups instead of one. It’s that we’re doing two lookups and the second lookup is fundamentally slower than the first.
Surely Someone Noticed This Before?
This has been noticed before. Unfortunately, with nobody using IPv6, the current slowness was an acceptable risk. Application vendors (namely browser vendors) have said, “this isn’t our problem, host resolution is the OS’s jobâ€.
The net result is that everyone knows about this flaw. But nobody fixed it. (Thank goodness for DNS Prefetching!)
Just last year, the “Happy Eyeballs†RFC was introduced which proposes a work around to this problem by racing connections against each other. This is an obvious idea, of course. I don’t know of anyone implementing this yet, but it is certainly something we’re talking about on the Chrome team.
What is The Operating System’s Job?
All browsers, be it Chrome or Firefox or IE, use the operating system to do DNS lookups. Observers often ask, “why doesn’t Chrome (or Firefox, or IE) have its own asynchronous DNS resolver?†The problem is that every operating system, from Windows to Linux to Mac has multiple name-resolution techniques, and resolving hostnames in the browser requires using them all, based on the user’s operating configuration. Examples of non-DNS resolvers include: NetBIOS/WINS, /etc/hosts files, and Yellow Pages. If the browser simply bypassed all of these an exclusively used DNS, some users would be completely broken.
If these DNS problems had been fixed at the OS layer, I wouldn’t be writing this blog post right now. But I don’t really blame Windows or Linux – nobody was turning this stuff on. Why should they shine a part of their product that nobody uses?
Lesson Learned: Only The Apps Can ‘Pull’ Protocol Changes
IPv6 deployment has been going on for over 10 years now, and there is no end in sight. The current plan (like IPv6’s break the internet day) is the same plan we’ve been doing for 10 years. When do we admit that the current plan to deploy IPv6 is simply never going to work?
A lesson learned from SPDY is that only the applications can drive protocol changes. The OS’s, bless their hearts, can only do so much and move too slowly to push new protocols. There is an inevitable chicken-and-egg problem where applications won’t use it because OS support is not there and OSes won’t optimize it because applications aren’t there.
The only solution is at the Application Layer – the browser. But that may be the best news of all, because it means that we can fix this! More to come…
Great article Mike! Very interesting findings on the impact of IPv6 on the performance of DNS lookup. I am looking forward to the rest of the posts in the series.
Also, I would highly recommend that you re-publish this post and future ones on CircleID, which is community about Internet infrastructure.
Drit
Please describe more about your test environment. I am seeing correct, parallel lookups on Linux that have virtually no increase in latency (about one ten-thousandth of a second at most). Tested Chrome 11, Firefox 3.5 and getaddrinfo() on Debian with eglibc 2.11.2.
I wish I lived in a world where networking bottlenecks could be measured in milliseconds.
In my quest for speed, I upgraded from one provider (that could deliver only 768kbps, but had virtually no downtime) to the only other provider in the area (which claims 3 Mbps, but has hours of down time per week)–for 2.5 times the price. Just the other night, I was able to stream a 20 minute YouTube video in a mere 60 minutes.
Another issue I’ve observed is that some AAAA queries just never get cached. Run “dig @8.8.8.8 http://www.etrade.com A” vs “dig @8.8.8.8 http://www.etrade.com AAAA”, and note that even if you do this over and over again, the AAAA query is always significantly slower.
Gah, please ignore the “http://” prefixes from my previous comment. They were inserted automatically by the server.
erm, http://www.etrade.com doesnt have a AAAA record
bohara@panther:~> dig http://www.etrade.com aaaa
; <> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 <> http://www.etrade.com aaaa
;; global options: printcmd
;; Got answer:
;; ->>HEADER< dig http://www.etrade.com a
; <> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 <> http://www.etrade.com a
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56311
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 0
;; QUESTION SECTION:
;www.etrade.com. IN A
;; ANSWER SECTION:
http://www.etrade.com. 300 IN A 198.93.34.21
;; AUTHORITY SECTION:
http://www.etrade.com. 3564 IN NS gsched5.etrade.com.
http://www.etrade.com. 3564 IN NS gsched7.etrade.com.
http://www.etrade.com. 3564 IN NS gsched4.etrade.com.
http://www.etrade.com. 3564 IN NS gsched8.etrade.com.
;; Query time: 201 msec
;; SERVER: 193.0.1.97#53(193.0.1.97)
;; WHEN: Thu May 19 13:27:26 2011
;; MSG SIZE rcvd: 136
I find this a bit hard to buy (entirely). More details please!
(you may like to read this somewhat outdated, but stil interesting document:
https://www.dns-oarc.net/files/workshop-2006/Toyono-StubIPv6.pdf)
Here’s a trace from my iMac (with Safari browser):
19:59:17.569446 IP imac.home.example.net.64968 > xs-eth0.home.example.net.domain: 58102+ A? webwereld.nl. (30)
19:59:17.569556 IP imac.home.example.net.63651 > xs-eth0.home.example.net.domain: 56498+ AAAA? webwereld.nl. (30)
19:59:17.620396 IP xs-eth0.home.example.net.domain > imac.home.example.net.64968: 58102 1/0/0 A 85.112.20.124 (46)
19:59:17.620650 IP xs-eth0.home.forfun.net.domain > imac.home.forfun.net.63651: 56498 0/1/0 (88)
As far asI can tell, both the the A reply and the empty reply for the AAAA query arrive at virtually the same time.
And it seems as if the AAAA queries leaves, before the answer to the A query has arrived, right?
Same for a host lookup that has an IPv6 address:
20:06:44.026331 IP imac.home.example.net.51979 > xs-eth0.home.example.net.domain: 52422+ A? ip.sidn.nl. (28)
20:06:44.026435 IP imac.home.example.net.61669 > xs-eth0.home.example.net.domain: 9915+ AAAA? ip.sidn.nl. (28)
20:06:44.058697 IP xs-eth0.home.example.net.domain > imac.home.example.net.51979: 52422 1/2/4 A 213.136.31.216 (168)
20:06:44.060610 IP xs-eth0.home.example.net.domain > imac.home.example.net.61669: 9915 1/2/4 AAAA 2001:7b8:c05::80:1 (180)
What is your opinion about this?
Please also explain why in your Linux example, the AAAA query comes first, instead of the A query, while you write that OS-es ‘basically do the same’.
Thank you.
Regards,
—
Marco
That last ‘ imac.home.forfun.net’ should read as ‘imac.home.example.net’.
I attempted to obfuscate some real life-details. I clearly failed.
@bohara: I confirmed @tedp.id.au’s results. Measure the time it takes to do the lookups. For me:
A record lookups: 19ms, 16ms, 7ms, 7ms, 7ms
AAAA record lookups: 62ms, 63ms, 32ms, 31ms, 27ms
I don’t offer an explanation for this, but observe that one is slower than the other.
@marco.davids – thanks for doing the test! Based on your trace, it is clear that the latest version of MacOS does do parallel lookups. Most Macs don’t 🙂 So in a few years, it will get better. We hope to add a test to chrome to measure real IPv6 vs IPv4 lookup times – so in the not too distant future we can use real, aggregate data rather than anecdotal lookups like yours and mine.
I did notice that Linux throws out the AAAA request before the A request. It doesn’t matter – what matters is if it is parallel or serial, and in my trace it was serial (although @tedp.id.au observed that some versions of linux are doing parallel as well). The getaddrinfo() API has to do both lookups, and then sort the IPv6 addresses to the top, and then return it to the application. So, which one starts first doesn’t matter.
Finally, I apologize to the OSes that have fixed this within the last year or so – it’s great!!! Sadly, my article still holds true for the 90+% of the world that is running Windows or older versions of the OS that don’t do them in parallel.
dig http://www.etrade.com a 0.00s user 0.03s system 4% cpu 0.665 total
dig http://www.etrade.com aaaa 0.00s user 0.00s system 1% cpu 0.289 total
aaaa lookup seems 3 times quicker to me, thats one a working dual stack machine with working v6 DNS servers.
as for OSX, it does multicast DNS queries and uses whichever A or AAAA gets returned first.
Any chance you could put a note in your article about the Linux behavior not matching what any of us in the comments have found? I don’t doubt that there could be *some* Linux distro that does sequential lookups, but it appears to be exceptional, and modern distros behave correctly with no discernible reduction in performance.
We’ll get real data from the field – if it shows that ipv6 is faster on the fleet of chrome machines out there, I’ll happily update with those notes. I remain highly skeptical 🙂
Pingback:Google Claims 30% Latency Reduction In Chrome | ConceivablyTech
OS X has been doing parallel lookups since Snow Leopard was released in 2009.
The reason I know this is because there was a bug in the implementation of mDNSResponder that caused any answers after the first one to be discarded. It was a nasty bug — which has been fixed — but OS X has been doing parallel resolution for a long time.
Also, I’m not sure that I buy your explanation of caching behavior for AAAA records. Don’t all major browsers request them, even if the machine doesn’t have IPv6 connectivity? From the point of view of a DNS resolver, I don’t see why these records would be any more eligible for eviction as compared to A records.
Pingback:IPv6 DNS Lookup Times « Mike's Lookout
My first experience with ipv6 was a disaster. I didn’t understand why some sites like google, hotmail, ebay, etc were slower with ipv6. Then I ran a packet capture. The full trace is here if you are curious.
The sites with SSL seems much slower than the other and google is now using https! Here is what happenned in firefox 15 in Windows XP SP3 with ipv6 when I ask for http://www.hotmail.com
I agree with the blog that yes the dns queries are a little bit slower.
There are 20 dns queries and they take 20 seconds to finish! During that 20 seconds nothing happen in my browser and it looks frozen.
All the queries are link to ocsp and CRL.
In firefox, click on tools, options, click in the advanced icon, then encryption tab, finally click on validation and uncheck “use…”
Then internet with ipv6 is fast again. Whose fault is it? I don’t know.
If you need ipv6 help to understand why you are applications are slower I can help you identify the source of the problem. I am a ipv6 guru and I can be reach at wedebugyou.
I did more research and this University helpdesk wrote the solution to this. In fact it’s not a solution it’s an ugly patch.
I will resume the solution. Many software relies on dns to fetch features. Some are not on ipv6 and the actual implemention is long to timeout and realize it’s not there…
So in firefox any version go to the address about:config
Search for the key network.dns.ipv4OnlyDomains and double-click on it.
You only need to add the ipv4 domains in this box as a comma space separated list. I posted one here so far and we can contribute. Send me an email and I will update it.
If you have slowness do a packet capture with tcpdump or wireshark and look for the dns queris AAAA that timeout and retry a lot.
Until this slowness issue in the OS or the apps are not fix we can still enjoy ipv6 with this work around.
Jean