Category: Technology

Guide to Bitcoin & Digital Asset Insurance

Mike 0 Comment bitcoin, Technology

Guide to Insurance

BitGo has been a pioneer in establishing the Bitcoin & Digital Asset Insurance capabilities for custodians dating back to 2015. Over the years, we’ve built the most comprehensive package available, which many exchanges and custodians have replicated with varying degrees of success. Whether you’re looking to buy digital asset insurance or whether you want to understand the options, here’s what you need to know!

tldr;

  • you can’t simply compare two coverage limits; not all policies cover the same risks
  • there isn’t enough global insurance coverage for all service providers and their assets under custody
  • cold storage & multi-signature technologies, combined with SOC audits and operational controls are the key to low insurance rates and more coverage

Coverage Limits
Most clients simply look for the coverage limit of a service provider. The insurance coverage limit is indicative of the financial health of the service provider in terms of their ability to afford insurance, and also the ability of that service provider to acquire insurance at affordable rates. Companies with stronger security models, audits, and controls always are able to carry more insurance because the insurance policy rates are significantly lower. Examples of risk-reductions leading to lower priced insurance include: cold storage, SOC-controls, multi-signature capabilities, and geographic dispersion of keys. The difference in rates can be significant. Hot storage insurance, such as what you get with MPC wallet technology, generally costs 1.75 – 3.00% per year! Cold storage wallet solutions, by contrast, generally carry rates significantly below 1%.

At BitGo, we have a $250M shared pool limit with excess insurance available to clients that purchase it. The excess cover currently employed by clients at BitGo is over $500M, making the total cover at BitGo $750M, the largest in the industry, and still growing.

Global Coverage Capacity
As you look for service providers, you need to be aware that the digital asset insurance market is pretty small. The exact size of total insurance is unknown, but is estimated to be around $5-7B of total coverage shared globally. Given that the market cap of crypto today is about $2.8T, that means insurance is only available to cover a tiny fraction – less than 1% – of all crypto available.

Worse, $5-7B is smaller than the amounts held at many custodian’s and exchanges. Even if BitGo or Coinbase bought 100% of the available insurance supply, we wouldn’t be able to cover all of the assets held in custody. On the surface, this sounds terrible. But there is good news: digital asset technology does provide security provisions which, if used properly, can stretch insurance over a larger pool of assets. For example, at BitGo, if you wanted to store $1B of Bitcoin with us, we wouldn’t store it all in one big wallet. Rather, we’d break it up into multiple wallets, of less than $200M each, each with independent keys. Further, we’d rebalance wallets over time as prices fluctuate, to minimize the chance of a single-wallet compromise that could exceed the insurance policy. Further, because client funds are generally segregated and split, most wallets remain dormant for very long periods of time. In fact, if there are deposits and withdrawals, we will generally only use one of those wallets for transactional needs to further reduce access to wallets that can otherwise be held with keys dormant.

As you select your custody provider, make sure they have good policies about how they segregate funds internally. Not all custodians segregate funds on-chain or enable rebalancing and other security protections which can extend insurance policies effectively.

Covered Risks
The most difficult part of evaluating any two service providers is understanding what their insurance policy actually protects against. Just because a company has insurance doesn’t mean that all assets are covered or even that the same risks are covered. Unfortunately, no two policies are alike!

Years ago, I heard from another company that their insurance policy rate was about 1/3rd the price of what BitGo was paying at the time. Of course I was interested in finding out how to reduce BitGo’s insurance costs, so we reached out to their broker right away. The broker was very straightforward and honest and quickly told me, “First off, you need to know that our policy is very good, but it doesn’t cover all risks. It won’t cover any hacks.” I was shell-shocked; how could any digital asset custodian possibly be interested in an insurance policy which doesn’t cover the one vector of risk that our clients care about most? Obviously BitGo did not switch to that low cost provider and stayed a better policy.

However, to this day, I know many digital asset companies are still using that “cheapo” insurance. They claim to have large amounts of insurance just like BitGo does. But theirs doesn’t cover hacks at all, while BitGo’s does. It’s so fundamentally different you can’t even compare.

When selecting your service provider, check to see what risks are protected. Some things to check:

  • insider theft
  • hacks
  • technology failure
  • geographic / natural disaster

Geographic risk coverage may sound unsual for digital assets, but remember that some custodians use single-signature wallets (not BitGo, of course!). This means that a natural disaster at the wrong location could lead to catastrophic loss. BitGo’s 100% multi-signature and threshold technology cold storage means that we can always store the 3 keys at least 1000 miles apart from each other. BitGo was the first digital asset insurance to carry this feature, and I believe that even today, BitGo is the only provider in the industry to offer it.

Quality of Underwriter
All of us want to ensure that a loss event does occur, that the insurance underwriter will pay the claim. Given that insurance policies in crypto tend to be measured in the 10’s or 100’s of millions of dollars, the underwriter matters a lot. One of the highest reputation underwriters is the Lloyd’s of London syndicates. These underwriters are known for their long-standing in the industry and ability to pay in the event of a loss event. While there are many underwriters out there, make sure you evaluate whether the coverage you think you’re buying is being provided by someone who will stand by their policy when you need it.

Beware of Additive Policies
Some service providers in the space use multiple service sub-providers, which may provide insurance. We’ve definitely seen marketing of policies where the sub-provider policies are simply added up, enabling the service provider to advertise a very large number. Remember, just because a service provider has a policy or a sub-provider policy, it does not mean that your assets are insured there. The additive policy might claim to have $500M of insurance from 3 different sub-providers. But if you store $500M yourself at that provider, it is unlikely each of the sub-providers would help with your recovery.

Beware of Fake Coverage
Some service providers will advertise a sub-provider’s policy as their own. Years ago, BitGo had a client, call him Alex, using BitGo for custodial and non-custodial wallets. At the time, Alex liked to tell his customers that he used BitGo, with $100M of insurance. But Alex wasn’t simply holding onto the assets in BitGo wallets, he was doing other things with his client’s assets, and very little was actually stored in BitGo. While it is true that any assets Alex held with BitGo in cold-storage were be covered if BitGo lost the funds, it did not mean that BitGo covered losses for Alex if funds were not stored at BitGo. After inquiries from Alex’s customers, we tried to get Alex to change his marketing, but could never come to agreement. Eventually, in 2020, we were forced to drop Alex as a client because of his misleading insurance claims. To this day, I do believe many service providers are less-than-forthcoming about what assets are stored with custodians like BitGo and covered by insurance. Make sure you clearly understand who holds the policy and which assets are insured. No insurance underwriter ever backs a policy protecting assets which are not in the direct care of the policy holder.

Hope this was helpful!

Why now is the best time to build in crypto

Mike 0 Comment bitcoin, Technology
BUIDL

We all have one thing in common. None of us knows how much time we have, and so for all of us time is our most precious resource. So the question is, what will you do with your precious time?

All great achievements take time. So, if you’re a developer who aspires to build something that’s going to have an impact on an industry or on the world, it’s going to take years of your time. It’s important to be intentional about what kind of impact you want to have.

I would submit to you that, right now, building for crypto, web3 and DeFi is probably the most impactful opportunity of our lifetimes. And it is starting right now. You can be part of the beginning of the financial system revolution.

I’ve been incredibly fortunate to have had the opportunity to work on a lot of high impact products. I joined Netscape just as the Internet–web 1–was being born. I quickly became an expert in the inner workings of the Internet and its protocols, and it changed everything in my career. Fast forward a few years and a few great start-ups later, and I was one of the first ten engineers on the project that would become Google Chrome, the web development platform that enabled web 2. Now, with BitGo, I’m building for web 3. 

But I got lucky. When I was starting my career, I didn’t think intentionally about where I could make an impact. To be honest, I was probably too young to realize that is what I wanted.

My first job was at Hewlett Packard. At the time I joined in the 1990s, it was still a great company, but probably past its peak. My job was fixing bugs within their MPE operating system on their legacy minicomputer line. It was a good place to start to learn. But I was a piece within a piece of a piece of the company. I don’t look back at that time as well spent.

Around the same time, the foundations of the Internet were being invented. Tim Berners-Lee had published his paper on the World Wide Web. At the University of Illinois Urbana-Champaign (UIUC), Marc Andreessen and his team were building a browser called Mosaic, and later would form the core development team that brought us Netscape. 

An enormous amount of engineering talent was focused on building the capability to connect people to information. I applied for a job at Netscape and joined just before they went public. My parents thought I was crazy. But of course, Netscape soon became a global phenomenon and changed the world. And their minds.

Netscape was a 180 degree turn away from what I’d been doing at HP. It was pure chaos, but for an ambitious developer, it opened my eyes to the unlimited potential of software. I had my official role, but I also spent a lot of time researching and prototyping ways to make web servers better and faster.

The work we did at Netscape was fast and furious. Many of the original UIUC team were now at Netscape, and all of us were in our 20s. The code they had written as college students, and now as first time professional software engineers, was beginning to have an impact on mainstream culture and continues to do so to this day. I recall driving on highway 101 in Silicon Valley with some of the team around that time, and we drove past one of the first ever billboards with an “http://” URL on it. It was a profound moment – these young engineers were seeing their work taking root in the mainstream for the first time.It was inspiring.

The impact of Netscape and the speed of innovation would eventually lead Marc, the co-founder and visionary behind Netscape to say, “software is eating the world,” a remark which has proven to be prophetic to this day. It’s software that gives us the ability to change an industry, and to improve over time. Netscape was one of the first pure software companies driving that kind of upheaval. 

I was really fortunate to get in on the ground floor of something that had such an impact. It set me on a completely different trajectory to love startup companies and continue to try to build new things. 

The Internet was already about 12 years old when I started at Netscape and we started to build software for the web–about the same age that Bitcoin is now–and its impact has changed the world. The impact of Bitcoin, crypto, and web3 will be even greater because it intersects with money. It’s no coincidence that Marc and Ben Horowitz, co-founder of Andressen Horowitz, run the largest web3 & crypto venture funds in the world.

It’s going to take longer to build because unlike the movement of information, the movement of money brings regulation with it. We are still in the first inning. 

Crypto projects stand at just under a trillion dollars in global market cap, and yet it has the opportunity to encompass all of finance. Every major bank, custodian, and exchange on the planet is talking about how ultimately all of finance will be digital assets. But it’s unlikely that the incumbent players are going to win this market. 

It’s a classic case of the Innovator’s Dilemma. This is the phenomenon where the big companies, with all of the resources eventually lose to small companies with not much more than an idea and grit. We see this pattern over and over again. Software reduces the cost of development so that small, creative, and fast moving teams can iterate quickly, using their software to eat their much larger rivals’ lunches.

The early iterations from these small innovators are usually ground breaking. But the early markets for them are small. Market leaders, with their massive distribution channels and seemingly infinite resources, invariably dismiss these early markets as too small to be a threat. They pan the innovators as immature, inexperienced neophytes. Unfazed, the innovators continue to iterate and iterate, and eventually become a substantial force.

As the innovators start to reach critical mass, the incumbent firms take interest. But it’s too late. They’re starting from zero, they’re not very nimble, and they just can’t catch up. 

Microsoft beating Netscape in the “browser wars” was a brief exception to this rule. Microsoft would have missed the internet era had it not been for Bill Gates’ technical leadership and vision. He penned his famous “Internet Tidal Wave” memo in May, 1995, and it was a seminal moment for Microsoft. Gates immediately deployed Microsoft’s vast resources, its software engineering expertise, and its operating system monopoly to build a “Netscape killer”. It seemed like it was already too late for Microsoft. But Netscape faltered. They didn’t know how to manage large teams and their software development lifecycle was atrocious. Microsoft, by contrast, had been building software in the large for decades and they knew exactly what to do. Internet Explorer soon became the dominant product in the market by 1999. Microsoft escaped the innovator’s dilemma for a moment, but they stopped innovating and eventually Chrome rose and clobbered IE.

You could fairly argue that Microsoft beat Netscape because they had the second mover advantage. They were quickly able to replicate the product and then put all of their marketing muscle behind it. 

But that’s unlikely to happen in crypto. It is true that all of the big banks and brokerage houses have taken notice of crypto and web3, But Microsoft was a software company, and it’s software that eats the world. Banks don’t have strong software teams. They’re not product builders. They’ve built fragile markets based on marketing and relationships that have a tendency to fail spectacularly – as we saw during  the Great Financial Crisis of 2008. Relationships don’t scale like software. So these incumbents are unlikely to be successful second movers.

Additionally, today’s software engineers stand on the shoulders of giants like never before. These giants built open source software, git, github, front-end frameworks, continuous integration systems, and more. Netscape struggled to scale with the tools we had at the time. Today, any good computer science graduate can run circles around Netscape’s neanderthal development processes.

This is why the crypto industry is the best place to be a developer noq. We are at a once-in–a-lifetime point in time where “software eats the financial system”. This is going to be the =fastest and most innovative change in any system, ever. 

There are three reasons why:

  • First, it is well known that the financial services sector has seen little technology and innovation for years. This is due to institutions building regulatory moats rather than competing with service and technology. As a result, we now have decades of pent-up demand for better technology and software capabilities in the banking and financial services sector;
  • Second, the advent of Bitcoin and blockchain technologies has enabled software change to finally cross the regulatory moat that has enabled banks to stagnate for so long;
  • Third, change is occuring at a global scale. We’ve spent the last 25 years connecting every corner of the globe to build communications and communities everywhere. The change in the financial sector is not just in the United States, it is everywhere.

This confluence of trends makes crypto and web3 the most innovative space in software ever. If you’re a developer thinking about what your career is going to look like five, ten, or twenty years from now, it’s going to include this technology. You have the same opportunity that I stumbled into back in 1993: the creation of a whole new system!

But our industry is not without flaws.

Think again about how you spend the time in your life and the impact you want to have.

For the first 20 years of my career, I never was interested in finance. Traditional finance builds nothing. The entire finance industry makes money off the productivity of others.it It has almost no direct impact on GDP. What little “service” it does provide could be easily replaced by software that can do it better, faster, and without lying, cheating or stealing. 

What got me excited initially about Bitcoin, and eventually digital assets in general, is that we can make a more fair financial system. We can bring safe money to places in the world that don’t have it today. We can expose fraud. We can enable privacy so that everyone can be safer and freer. These are worthy of my time.  This is what I’m here to build. I could do this for a decade, or I could do it for a lifetime. 

Think about how you’ll reflect on your time on this earth when you’re at the end of your career. Wealth is great, if you can get it. But will you have built something of substance or value? Or will you have merely garnered riches by enabling speculators and gamblers?  

As software developers, you have power. You have power that bankers don’t have. You have the power to build. You get to decide what you build, and how you build it. Further, most software developers I know are mission driven. They are here to create and have impact.

Thus, because we are dealing with money, we have a greater responsibility as developers than most other sectors. We need to think about how our mistakes could impact people. Could it hurt their security? Could it hurt their savings? Could it enable governments to oppress or harm people? 

We know about cyber security breaches. We know about privacy concerns. We know about threat models and secure coding practices. But do we employ them? Or are we here to make a quick buck?

We can and will move quickly. But we must also move responsibly. The opportunity is huge. Our fortunes, and the fortunes of the world hinge on our next few iterations in all things crypto and all things web3.

The Reserve Currency of the World Will Be Digital Currency

Mike 0 Comment bitcoin, Editorial, Politics, Technology

This week, PayPal announced it is leaving the Libra alliance.  While that may sound like a vote against Libra, it’s more likely a vote against the dollar.  PayPal was just approved as China’s first western payment processor, and its likely that PayPal is simply betting that China’s digital currency is more likely to win than Facebook’s.

Today, US Dollars are the world’s dominant reserve currency, representing 62% of all reserves, while the RMB accounts for a mere 2%.  Despite having the globe’s 2nd largest economy by GDP, China’s RMB has only became reserve currency 3 years ago in October, 2016 and it has grown from 0% to 2% in a very short time. 

Meanwhile, in digital currency, American politicians are fighting digital.  Democrats, such as Senator Maxine Waters, and Republicans, such as President Donald Trump, both see Facebook’s Libra as a threat to the US Dollar.  Trump goes so far as to say that all digital currencies are an affront to the USD.  At the same time, China’s leadership is ready to embrace digital currency and is accelerating projects to create a government-backed digital currency.  Who’s right here? America sees digital as a threat while China sees digital as its opportunity to break out.

Frankly, digital currencies are unstoppable.  We have a global economy, but we don’t have a global currency.  Digital currencies, which can move globally with zero friction and zero counterparty risk are the first opportunity we’ve had to build a truly global currency.  While the US should use its position as the dominant reserve to propel digital currencies forward, it instead sees it as a threat and is attempting to block it. But the US has no jurisdiction globally, and its competitors would love nothing more than to displace the dollar.  As such, China rightly recognizes that digital currency is just what it has been waiting for: a global distribution vehicle for RMB.

If the US can’t get its act together soon, it will lose.  Crypto-currency is the future. As Marc Andreesen once said, “software is eating the world.“

Why I Would Not Participate in a MPC Wallet

Mike 0 Comment bitcoin, Technology , , ,

The personal liabilities associated with multi-party computation (MPC) based wallets are so great I don’t see how I (or anyone) could ever participate in a MPC wallet.

No Accountability

The core problem with MPC is an architectural one. While MPC does create a mechanism whereby multiple people can each hold independent parts of a key to eliminate single points of failure, MPC fails to offer any accountability about who participated in the signing of a transaction.

Imagine you create a 4-of-7 MPC wallet with 7 people participating, and 4 required to authorize a transaction. What if, unbeknownst to you, 4 of the other people holding key parts in the MPC wallet decide to steal the money? Because MPC does not offer signature accountability, no one can be certain who participated in the transaction.  As such, even though you had nothing to do with the crime, you’re now a suspect, and it may take months or years to clear your good name.

Co-Signers Make MPC Even Worse

Vendors offering MPC services and co-signing dismiss this vulnerability and claim, “don’t worry, we keep track of who participated and will log all accesses to the signing process”.  In other words, even though no one can determine who participated in the transaction from the signature itself, the vendors claim that they know the answer within their application logs. Thinking about this carefully, you’ll realize this makes the vulnerability even more severe.

With the vendor as a co-signer, you can now imagine the same attack scenario as above where 4 of the other participants on the wallet collude to steal the money.  In this case, however, imagine one of the perpetrators is a rogue employee at the MPC vendor itself. In this scenario, you have no protection that the MPC vendor isn’t modifying its application logs and data. In addition having already been a suspect, the MPC rogue employee or vendor can now frame you for the crime. How would you defend yourself in this scenario?  They hold all the cards, the data, the logs, and the technology.  Unless you’re a cryptography expert, it will be extremely difficult to defend against them.

Conclusion

MPC vendors forget that accountability is a critical part of security, trust, and safety in a multi-user system. Participants on MPC wallets need to be very careful that they can fully trust all of their MPC wallet co-participants. This may not seem like a large risk if your wallet balances are small. But these vendors are encouraging MPC for protecting billions of dollars of assets. 

Multi-signature systems, by contrast, offer all of the benefits that MPC systems offer, but without any ambiguity of accountability.  With a multi-signature system, everyone on the blockchain can publicly see that you did not participate in the transaction without a shadow of a doubt.

I don’t see why anyone participating in the security of assets would even consider using MPC without multi-signature.  The personal risk for the users of the MPC system is massive, and is simply beyond tolerances as the asset values go up.

Proprietary Cryptography

Mike 2 Comments bitcoin, Technology

One of the best things about the growth of Bitcoin is how it has propelled research and development in cryptography. What was once a relatively sleepy field of computer science has now become one of the most popular areas of study.  There is no doubt that this additional research will yield great advances in the coming decades. But cryptography is unique from other computer science disciplines, in that there is no margin for error – especially if that cryptography is being used to secure money or digital assets. Unfortunately, the growth of Bitcoin has also fueled a new wave of rushed cryptography.  Rushed cryptography is brand new cryptographic technology that hasn’t had sufficient peer review or test, yet is being promoted as the new panacea to all your hacking woes.

The creators of rushed cryptography always know that they rushed it.  They know they haven’t done sufficient testing or peer review. Testing takes months to years and peer review takes years to decades.  Excited to launch products with their new technology, combined with a little hubris and a little ambition, rushed cryptographers use their new algorithms prematurely. While they make bold claims and brag about the awesomeness of their creation, internally, the rushed cryptographer is actually full of fear – fear that someone will find a bug, a hole, or a problem before they do.  To prevent this from happening, they fall back on the oldest trick in the book: they make it proprietary.

What is proprietary cryptography?  Nobody knows except the creator – the same one that is now trying to sell you his product. The creator says they tested it.  They hired PhD’s, experts and mathematicians to attest they did a great job. They hired security auditors and code reviewers. But did they?  How can you know? How can you possibly use this to secure assets worth millions?

OWASP (the Open Web Application Security Project) has this to say about proprietary cryptography: “Proprietary encryption algorithms are not to be trusted as they typically rely on ‘security through obscurity’ and not sound mathematics. These algorithms should be avoided if possible.”

Remember Schneier’s Law: “Anyone, from the most clueless amateur to the best cryptographer, can create an algorithm that he himself can’t break.”

It is an exciting time for cryptography, computer science, and digital assets.  But one of the best things about Bitcoin is that it relies on stable, steady, known algorithms.  This conservative development helps the system, builds trust, and is known to be secure. To those that are rushing new crypto, don’t forget peer review and open source implementations: this is money!

Encrypt It All

mike 0 Comment Technology

I’ve had several of my non-technical friends ask me about Apple’s recent plan to encrypt everything on your smart phone. Of course, Google has been moving to this plan for quite some time already. But the recent announcement has led to the government’s formal claim that this is a threat to national security. What is the truth?

Court Orders are not Required
The first problem facing Apple, Google, and others is that even in America, many of the agencies requesting private data do so without a court order of any kind. Most American citizens think that the US would never do this – but in fact it happens almost every day. As you can see from the Google Transparency Report, there are nearly twice as many requests without court orders as requests with court orders. If you were Google, how would you decide when a request needs to be obeyed and when it should be rejected? Is there any policy that Google, or any company, could possibly apply that would be correct?

Unfortunately, government agencies believe they can simply make these requests and expect the company to comply. If the company does not comply, the company is threatened by that agency – sometimes with threats of jailtime under laws that appear to undermine our constitution. How does the company know whether a request should be obeyed when there is no court order? How can a company remain objective when government agencies are not required to follow due process?

Too Many Governments
The other sad fact is that there are simply too many governments and too many laws. Tech companies are global and need to respect global law. Unfortunately, this means fielding requests from all over the world. How much legal time should a company expect to spend answering requests from agencies around the world? How do you know if a request is authentic? What if you received a request for private data, and complied, but it turned out to be a fraudulent request? Attempting to comply could lead to more damage than not trying to comply. Unfortunately, our governments do nothing to help keep us, the citizens, safe from fraudulent requests. They simply demand information as though they are entitled.

The Only Solution Is Not To Know
The result is that all companies will eventually opt for the only solution that makes sense – encrypt everything. In the wise words of Commander Klink, “I know nothing” is the only way to deal with these requests. If you don’t hold the keys to your customer’s information, you can’t possibly give it to anyone. Unfortunately, since government agencies make too many subjective and illegal requests for information, companies must defend themselves by simply dropping all knowledge. This is the approach we take at BitGo. We will comply with all court-ordered requests, but we know absolutely nothing, so don’t bother asking.

There is a silver lining to this for users, which is that their data will be more private and more secure. But it should be a warning to all of us, even here in America. When your government can spy on you illegally, you do not live in “the land of the free”.

Imagine A World With No Door Handles

mike 0 Comment Technology

It’s Saturday morning. You’ve been dying to get that latest iPhone/iMac/iWatch/iFad. So you head down to the Apple Store. When you arrive, the doors are closed. You try to push the door, but the door won’t open. It looks like people are inside, but you can’t figure out how to get in.

Suddenly, Steve Jobs walks up and asks, “Hi, may I help you?”

“Whoa! Steve Jobs! What an honor to meet you! I was just trying to get into your store, but there are no handles on the doors.”

Steve says, “Oh, of course. We decided to simplify the design of the door, giving it a sleek, elegant new look. Those old fashioned handles were just plain ugly. How do you like our new doors?”

“I guess they look okay. But I can’t get in.”

“Of course you can, you just have to tell the door to open, like this… Open, door.” Calmly, he waits a moment, but nothing happens.

Surprised, Steve tries again, “Open door.” A little louder – “Open door!” “Door, o-pen!” “O-pen Do-or.”

“Hmm…,” Steve shrugs, “the door seems to be having trouble right now. Oooo-PEN DOOR!” And at last, the doors open.

“Ah, there, you see! Had a small glitch, but isn’t that amazing?”

—–

Unfortunately this silly story is indicative of a dangerous pattern we’re seeing in software and hardware today — the rise of Design before Function. A great aesthetic look is great, and there are some instances where it can be more important than minor function. But in general, a great product trumps a great design any day of the week. From Craigslist to Ebay to Amazon to Google – products that people love are products that work more than products that are ‘designed’.

This morning, I tried to add a new keyboard to my iMac. What could be simpler, right? Well, it turns out the Mac simply can’t do it. The Apple “Genius” wants be to lug my entire computer into the Apple store to figure it out. You’ve got to be kidding me, right? In the quest for the elegant look, Apple removed the basic controls for setting up your computer. Now, I have a $2000 brick that needs to be taken to the store. This problem was created solely by design. They were more interested in removing buttons from the back of the computer than they were about making sure basic tasks could be easily done. Ah, but Macs are so easy to use, right?

Custom Protocols Everywhere

mike 1 Comment Technology

I’m here at the IETF discussing HTTP/2.0 (was SPDY).

One of the most interesting developments to me is that the rate of protocol evolution is about to step onto a rocket ship. In the past, we’ve been dependent on standards, browsers, and servers to all simultaneously move to new protocols. It takes about 15 years, in a good case! But mobile apps are cracking this open. With native code, instead of relying on the browser’s network stack, you just include your client library (imagine a facebook client library which speaks a custom protocol optimized just for facebook). Since these apps are not doing general purpose web surfing, they’re only connecting to their own servers. If you own the client, and you own the server, who needs a standard?

This is going to lead to massive innovation in protocols. Companies can invent new paradigms and algorithms, and then either keep these to themselves or kick them back to the IETF for later. From what I hear, SPDY sounds like a promising starting point for these apps, and it will lead to splintering. But you don’t have to wait for Google, Microsoft, and Firefox to agree on protocols anymore! As Will Chan puts it, it’s like HTML for HTTP – everyone can change it in practically real time.

The one last challenge is getting your protocol to work across the internet, where proxies, intermediaries, carriers, and other middlemen are always trying to trip you up. But fortunately, we have an answer for that too – tunnel it all through SSL, which you should be doing as a best practice anyway.

Visualizing SPDY vs HTTP

mike 3 Comments Technology

When we were developing SPDY, we wanted to know if SPDY was always faster or just sometimes faster than HTTP. The following chart is what convinced us that SPDY is indeed almost always faster. I didn’t publish this at that time because I didn’t like looking at web performance as a single number. “Is it faster” includes many variables, such as the network simulated (bandwidth & latency), the packet loss rate, the content chosen, and the measurement (total PLT, first PLT, second PLT, or time-to-first-render). You’d really want a whole stack of these charts, rather than just a single combination.

What I like about this chart is how easy it is to compare two protocols. If the two protocols are identical in performance, all points would be on the midline (red). If one protocol is slower, then the points will fall closer to that axis.

SPDY vs HTTP PLT

Notes about this test:

  • Used a static copy of the Alexa Top-300 websites (excluded porn sites), full content, unchanged
  • Simulated 2Mbps download, 386Kbps upload, 100ms RTT, 0% packet loss
  • No SSL
  • The average speedup was ~40% on this test
  • This test predates CWND changes in the kernel
  • Server was linux, client was chrome on Windows 7