Custom Protocols Everywhere

I’m here at the IETF discussing HTTP/2.0 (was SPDY).

One of the most interesting developments to me is that the rate of protocol evolution is about to step onto a rocket ship. In the past, we’ve been dependent on standards, browsers, and servers to all simultaneously move to new protocols. It takes about 15 years, in a good case! But mobile apps are cracking this open. With native code, instead of relying on the browser’s network stack, you just include your client library (imagine a facebook client library which speaks a custom protocol optimized just for facebook). Since these apps are not doing general purpose web surfing, they’re only connecting to their own servers. If you own the client, and you own the server, who needs a standard?

This is going to lead to massive innovation in protocols. Companies can invent new paradigms and algorithms, and then either keep these to themselves or kick them back to the IETF for later. From what I hear, SPDY sounds like a promising starting point for these apps, and it will lead to splintering. But you don’t have to wait for Google, Microsoft, and Firefox to agree on protocols anymore! As Will Chan puts it, it’s like HTML for HTTP – everyone can change it in practically real time.

The one last challenge is getting your protocol to work across the internet, where proxies, intermediaries, carriers, and other middlemen are always trying to trip you up. But fortunately, we have an answer for that too – tunnel it all through SSL, which you should be doing as a best practice anyway.

I will donate $1,000,000 to the US Govt This Year

The US debt is a big problem (currently ~$16T, or about ~$50K per person). I’m willing to do my share to help pay it off. To do this, I need the government to commit to a sustainable plan so that I’m not just throwing good money after bad.

Thus I make this promise:

If Washington can make spending match tax receipts, I will donate an extra $1,000,000 to the USA to help pay down the debt.

The Web only Works Thanks to Reload… (and why the mobile web fails)

When you build a mobile app that uses the network, it is instantly clear that your app needs to be robust against all sorts of network failures:

  • network completely down
  • network transitioning from WiFi to 3G
  • network insanely slow (EDGE!)
  • network timeouts – is 5s long enough to wait? 10s? 30?
  • network radio warmup is slow
  • what happens if your app is terminated before finishing a critical network request?
  • etc…
  • Dealing with these is hard, but not impossible. Applications retry at various levels trading off battery life and user perceived performance all the time. After enough work, you can make the app functional.

    But if you try to write an app using HTML5, how do you do this?

    You can’t.

    The web simply isn’t designed for partial network failures on a web page. Web pages are comprised of hundreds of subresources from multiple sites these days. What happens when CSS #5 out of 32 resources fails to load? What happens when you can’t connect to even though you already loaded the main content from Does your application even know? Generally not. You can trap for some sorts of errors; but the browser will NOT automatically retry on any of these failures for you. Most likely you’ll be left with a web page which renders incorrectly, hangs forever, or throws javascript errors on the whole page because a critical set of code just never got loaded.

    Of course, these problems can happen on your desktop, too. But they generally don’t happen as often. And when they do occur, every user easily becomes his own network administrator thanks to the web browser’s handy dandy “reload” button. How many times have you drummed your fingers for a few seconds before reloading a page? Probably a lot! But on mobile, network errors occur *all* the time. Do mobile apps have ‘reload’ buttons? Generally not – users are becoming quite accustomed to apps which can handle their own errors gracefully.

    Sadly, I think this is one more nail in the coffin against HTML5 on mobile. Browsers need to be completely overhauled to properly deal with network errors and retries before HTML5 can be a serious contender to native applications.

    I don’t listen to voicemail

    For a long time, my voicemail wasn’t set up. People got mad at me.

    So I setup my voicemail, and now it has a greeting. But I don’t listen to messages. People get mad at me again.

    The trouble with voicemail is its just too inefficient. I have to dial-in, wait for it to connect, listen to the rambling, and then act on it. The whole process is quite maddening – 20-30 seconds each! I know that at this very moment I have voicemails from my sister and brother, and I even have spare time. But the mere thought of the painful process makes me not do it. I’d rather write a whole blog post about how I hate listening to voicemail.

    Hmm… I wonder how many messages I’ve got queued up?

    Origin http://localhost is not allowed by Access-Control-Allow-Origin

    I had a frustrating day. I’m writing some simple ajax/xhr tests, and I can’t get the browser to issue my requests:

    Origin http://localhost is not allowed by Access-Control-Allow-Origin

    Searching on Google, it’s clear that lots of other people are having the same problem, but nobody has an answer – the browser nazis are being overzealous, protecting us from localhost – really?

    Fortunately, chrome has a workaround. Use this commandline:

    chrome.exe –disable-web-security

    And you can get your job done.

    Fire Your Lawyer

    Just spent 3 days waiting for turnaround on a contract where the other party’s counsel did nothing but twiddle the bits on the confidentiality clause. What a waste of time, productivity, and legal bills.

    Don’t get me wrong – the other lawyer is not the only one at fault here. My lawyer is equally to blame.

    What the hell? Why is it that my lawyer, and the other lawyer both thought they needed to reinvent a confidentiality clause? Can’t we just create one, have it ratified by the courts and live with it? Why do two lawyers get to rack up wages at $300/hr to modify 2 sentences that they’ve modified a thousand times before?

    Lawyers who don’t work on fixing this problem are the problem. They all need to get together, settle this, and stop raping America.

    Honda Civic Hybrid Lawsuit

    Heather Peters has become semi-famous this week because of her small-claims suit against Honda. Honda reached a class-action settlement because the gas mileage of the Civic is simply not up to par. This matches my own experience. Aside from the first few months owning the car where I was both actively hypermiling and also driving exclusively highway miles, the gas mileage in my car has been disappointing. My hybrid gets ~38mpg right now, a far cry worse than the claimed 50mpg that I should be getting.

    The class action lawsuit was a joke. The lawyers walked away with millions, while the consumers were compensated with $100 and a video. And each Civic Hybrid consumer has thousands of dollars in damages. Worse, the settlement bars anyone who even complained about the settlement from receiving the settlement. I complained, of course, and my punishment is that I didn’t get the $100.

    Since there seems to be renewed interest in this whole affair, here is the full text of the letter I submitted to the court regarding the class action suit back in December 2009.


    JOHN TRUE, et al.,
    Plaintiffs, Case No.: 5:07-cv-00287-VAP-OP





    I am a member of the settlement class for Case No. 5:07-cv-00287-VAP-OP (True vs. Honda Motor Company Inc.). This letter officially declares my objection to the settlement on the grounds that this settlement is not in the best interest of the members of the settlement class.

    This lawsuit was brought against American Honda Motors (AMH) because the fuel efficiency of their cars does not match the advertised claims. Evidence has been presented that the cars were advertised to achieve 50MPG while only actually achieving 32MPG for many drivers. Drivers of these cars often purchase them specifically because of the alleged low cost of operation. When the gas mileage fails to achieve advertised claims, the consumer is left to pay the difference. I believe the difference is an accurate representation of the damages caused by AMH.

    The Honda Civic warranties apply to various parts of the car for 50,000 to 150,000 miles, and consumers are sold the car with the expectation that 100,000 miles is reasonable
    . Gasoline in California has averaged ~$3.20 per gallon over the last 3 years
    . Using these figures, if the car only achieved 32MPG instead of 50MPG, the per-consumer damages are ~$3,600.

    I have 4 objections to this settlement.

    Objection #1: Failure to correct damages for the settlement class
    Given the magnitude of damages to consumers, the settlement offer simply is not in the best interest of the settlement class. The settlement reimburses the members of the class for less than 2% of the damages incurred.

    In the settlement, members are given 3 options:
    a) Sell your hybrid, and get a $1000 coupon on another Honda (not a hybrid).
    b) Give your hybrid to a family member, get a $500 coupon on another Honda (not a hybrid).
    c) Get a $100 check.
    All 3 options include a video for how to get better gas mileage.

    Even if the settlement member wanted a new car (which many do not), the $1,000 rebate still would still be less than one-third of the damages incurred due to unexpectedly low cost-of-operation.

    But worse, for most consumers, the $100 settlement is so small that it does not offset the damages at all. At the same time, this settlement greatly reduces the consumer’s claims against Honda, and may even void the consumers current warranty. Why would any consumer sell this privilege for a mere $100? (see the second objection #2).

    Objection #2: Overreaching settlement releases
    As part of this settlement, all members of the settlement class release all claims against AMH regarding anything related to the fuel economy of the vehicle. Specifically, the settlement covers anything

    “involving, based on, relating to, arising out of or in any way connected with, directly or indirectly, the advertising of the fuel economy or m.p.g. of the HCH…”

    With regard to existing warranties, it states

    “Nothing in this Agreement shall be interpreted to modify or diminish the manufacturer’s limited warranty with respect to a Class Vehicle; provided, however, that any such claim of breach of any warranty or any extended warranty based on the advertising or representations made by AHM with respect to fuel economy, mileage or m.p.g are in fact released.”

    This clause is completely overreaching! For instance, Honda warrants the batteries in the vehicle for 50-150K miles. When I was sold the vehicle, the salesperson told me that the batteries would not need replacement for 150K miles. However, as the batteries get old (much like the batteries in your cellphone), they hold their charge for diminishing amounts of time. The car will still operate just fine even with diminished battery life, but the MPG rating will drop significantly! I fear that if I accept this settlement, and then I go to Honda to get help with replacement of my previously-under-warranty batteries, I will no longer have a claim. Honda will be able to say that my car is working fine, and that the battery problem related to MPG was Released as part of this settlement.

    As such, this contract specifically DOES alter my warranty, despite claims to the contrary.

    Objection #3: The retaliatory nature of the settlement against its members.
    As part of this settlement, any member of the settlement class that objects to the settlement is specifically barred from collecting any benefits if the settlement is accepted by the court. This clause is clearly not in the best interest of the settlement class, and is actually retaliatory in nature.

    It is clear to me that the damages I have personally incurred due to AMH’s wrongful advertising are far in excess of the $100 settlement. Purchasing a new Honda has no appeal to me. I do not want to purchase a new car from Honda at any time in the future, and even if I did, I would like to get a hybrid. As such, options (a) and (b) simply do not make any sense.

    But the negotiators of this settlement have defined only 3 options, and I am required to either accept them or be ignored from any settlement result. This clearly only serves the interest of the legal staff and does not serve the interest of the plaintiffs.

    Objection #4: Negotiation of excessive attorney compensation at the expense of the settlement members
    I recognize that the attorneys have worked hard on this case for the last year. While they are pushing for a quick settlement which will enable them to receive a hefty payout of nearly 3 million dollars, they have failed to negotiate a settlement which actually addresses the original problem in the claim. If they believe that settlement members would benefit by accepting a plan which voids their warranty and only recoups 2% of the damages, then the attorney fees should also be diminished – I propose they should be set to 2% of the requested fees, or $92,187.

    My Status in the Settlement Class
    The following responses are the required responses as per the proposed settlement in progress.

    (i) Name address and phone number
    Michael Belshe

    (iii) Grounds for objection
    Stated above.
    (iv) Proof
    Stated above.
    (v) No persons will be called to testify
    (vi) I do intend to appear at the Final Approval Hearing, schedule permitting.
    (vii) I have appeared in no other cases as a settlement objector or counsel in the past 5 years.

    I have 5 recommendations for the settlement:
    (a) The negotiators should compute a reasonable figure for damages to each consumer. (I computed $3,600).
    (b) The negotiators should negotiate a settlement for at least 50% of those damages.
    (c) Objecting to a proposal must not cause members to be excluded from a settlement.
    (d) The settlement must re-address the warranty voiding effects mentioned above.
    (e) The attorney fees should be in-line with what settlement members receive in the settlement. If the settlement members recoup 50% of their damages, then the attorneys should only collect 50% of their $2.95M fee as well.


    Michael A Belshe

    Fixing the CEO Pay Problem

    I’m generally a pretty free-market kind of guy. But, when it comes to CEO pay, there is no doubt in my mind that America is screwed up and that the free market is failing us. This isn’t the biggest of our problems, but it raises unnecessary doubt about “corporate greed” and about the livelihood of the American Dream.

    If you don’t believe me, check out some of the compensation paid to CEOs of companies that are losing massive amounts of money:

    • Aubrey McClendon, Chesapeake Energy, paid $18.6M while the company lost $5.8B.
    • Carol Bartz, Yahoo, paid $39.0M in the same year she’s fired.
    • Timothy Armour, Janus Capital, paid $11.4M while the company lost $757.1M.
    • Rupert Murdoch, News Corp, paid $18.0M while the company lost $3.4B.
    • Robert Stevens, Lockheed Martin, paid $21.7M while the company lost $3.0B.
    • Daniel Hesse, Sprint, paid $10.3M while the company lost $2.4B.
    • Gregory Brown/Sanjay Jha, Motorola, paid $11.7M while the company lost $111M.
    • Ronald Hovsepian, Novell, paid $5.2M while the company lost $214.6M.
    • William Klesse, Valero, paid $11.3M while the company lost $353M.
    • Klaus Kleinfeld, Alcoa, paid $14.3M while the company lost $985M.
    • Ahmad Chatila, MEMC Electronic Materials, paid $16.7M while the company lost -$68.3M.
    • The list goes on and on…

    Still not convinced? Why do CEOs get golden parachutes? Why did Leo Apotheker get paid $25M after getting fired 11mos into the job? Do you get one? It makes no sense to ever have a guaranteed payout even if you screw up.

    Mark Cuban once again puts this in perspective by demonstrating that the risk-reward for CEOs is out of whack.

    Fortunately, it is easy to fix.

    The free market should remain free. If a company wants to pay a CEO $50M in advance, they are free to do so. But the Board of Directors, whose sole responsibility is to the shareholders best interests, needs to be able to prove that such a plan is good for the shareholders. If not, the Directors need to be held personally liable.

    I’d like to see the SEC adopt new rules about executive pay – including any form of guaranteed pay, pay for non-performance, pay while the company is losing money, or pay for early termination. These rules should outline a very strict and narrow definition for when such compensation would be “good for shareholders”. Common sense should win out here, and the right answer is “almost never”. We all know that if an employee isn’t working out you should fire them with impunity. CEO’s are no exception.

    As for the CEOs that are already beneficiaries of guaranteed payouts – if they have any character at all, they should forfeit these benefits and ask their Board of Directors to rework their compensation to something in line with what the rest of the company gets.

    SPDY of the Future Might Blow Your Mind Today

    This post is definitely for protocol geeks.

    SPDY has been up and running in the “basic case” at Google for some time now. But I never wrote publicly about some wicked cool possibilities for SPDY in the future. (Much to my surprise, it may be that someone is doing them today already!)

    To start this discussion, lets consider how the web basically works today. In this scenario, we’ve got a browser with 3 tabs open:

    As you can see, these pages use a tremendous number concurrent connections. This pattern has been measured both with Firefox and also with Chrome. Many mobile browsers today cap the connections at lower levels due to hardware constraints, but their desktop counterparts generally don’t because the only way to get true parallelism with HTTP is to open lots of connections. The HTTPArchive adds more good data into the mix, showing that an average web page today will use data from 12 different domains.

    Each of these connections needs a separate handshake to the server. Each of these connections occupies a slot in your ISP’s NAT table. Each of these connections needs to warm up the TCP SlowStart algorithm independently (Slow Start is how TCP learns how much data your Internet connection can handle). Eventually, the connections feed out onto the internet and on to the sites you’re visiting. Its impressive this system works very well at all, for it is certainly not a very inefficient use of TCP. Jim Gettys, one of the authors of HTTP has observed these inefficiencies and written about the effects of HTTP’s connection management with ‘bufferbloat’.

    SPDY of Today

    One first step to reduce connection load is to migrate sites to SPDY. SPDY resides side by side with HTTP, so not everyone needs to move to SPDY at the same time. But for those pages that do move to SPDY, they’ll have reduced page load times and transmitted with always-on security. On top of that, these pages are much gentler on the the network too. Suddenly those 30-75 connections per page evaporate into only 7 or 8 connections per page (a little less than one per domain). For large site operators, this can have a radical effect on overall network behavior. Note that early next year, when Firefox joins Chrome implementing SPDY, more than 50% of users will be able to access your site using SPDY.

    SPDY of the Future

    Despite its coolness, there is an aspect of SPDY that doesn’t get much press yet (because nobody is doing it). Kudos for Amazon’s Kindle Fire for inspiring me to write about it. I spent a fair amount of time running network traces of the Kindle Fire, and I honestly don’t know quite what they’re doing yet. I hope to learn more about it soon. But based on what I’ve seen so far, it’s clear to me that they’re taking SPDY far beyond where Chrome or Firefox can.

    The big drawback of the previous picture of SPDY is that it requires sites to individually switch to SPDY. This is advantageous from a migration point of view, but it means it will take a long time to roll out everywhere. But, if you’re willing to use a SPDY gateway for all of your traffic, a new door opens. Could mobile operators and carriers do this today? You bet!

    Check out the next picture of a SPDY browser with a SPDY gateway. Because SPDY can multiplex many connections, the browser can now put literally EVERY request onto a single SPDY connection. Now, any time the browser needs to fetch a request, it can send the request right away, without needing to do a DNS lookup, or a TCP handshake, or even an SSL handshake. On top of that, every request is secure, not just those that go to SSL sites.

    Wow! This is really incredible. They’ve just taken that massive ugly problem of ~200 connections to the device and turned it into 1! If your socks aren’t rolling up and down right now, I’m really not sure what would ever get you excited. To me, this is really exciting stuff.

    Some of you might correctly observe that we still end up with a lot of connections out the other end (past the SPDY gateway). But keep in mind that the bottleneck of the network today is the “last mile” – the last mile to your house. Network bandwidth and latencies are orders of magnitude faster on the general Internet than they are during that last mile to your house. Enabling SPDY on that link is the most important of them all. And the potential network efficiency gains here are huge for the mobile operators and ISPs. Because latencies are better on the open internet, it should still yield reduced traffic on the other side – but this is purely theoretical. I haven’t seen any measure of it yet. Maybe Amazon knows :-)

    More Future SPDY

    Finally, as an exercise to the reader, I’ll leave it to you to imagine the possibilities of SPDY in light of multiplexing many sites, each with their own end-to-end encryption. In the diagram above, SSL is still end-to-end, so that starting a SSL conversation still requires a few round trips. But maybe we can do even better….