On Shortcuts

Startups are often kick-started with a prototype. Perhaps not much more than tape and baling wire, this prototype can get the company users, noticed, funding.

Success is when the moment of true danger is, and the test of top-notch engineering and product leadership: Success is when you need to grow and evolve your system.

If you have the luxury to throw away your initial code and start from scratch, you’re in an unusually good position. If you aren’t, then the decisions made at the prototype stage have a profound impact on your forward velocity. A badly written prototype, full of cut corners and shortcuts is going to severely impair your ability to write new features, to find and fix bugs, to write a world-class test system.

The key is to know where to take shortcuts, and this applies even beyond the prototype stage: Never, ever take shortcuts on architectural correctness or code quality. Take shortcuts on features, on configurability, on performance. Don’t make every part of your system configurable, but architect it so that it can be. Don’t try to write a massively complex sharding system – rely on a single, robust database with a hot replica. Don’t attempt to give users three ways of doing anything in your system; focus on the path of least resistance.

At each step of the way, build just enough code to validate assumptions and test ideas, but build it right.

On SaaS and the Law of Large Numbers

Among our design mantras is that everything that can go wrong, will go wrong.

Is the concurrency model not perfect? Under the onslought of 10,000 hits per second, it’ll break.
Is the caching model not thought through completely? It too, will break.
Is there a single point of failure anywhere in the system? It will bring your system down.

We’ve learned to be, and to hire, perfectionists, because shortcuts do not work in a sophisticated, high volume environment. We’ve learned that every solution must be thought through and peer reviewed because any sufficiently complex system cannot be fully understood by a single person. We deploy at least two (typically four) instances of anything.

We are moving towards a model where the system is so loosely coupled that we’ll be able to shut down every single one of our database servers, and have the system continue running with no visible interruptions or loss of data. We can already shut down any web server with no visible effect to the system, as well as seamlessly launch new instances.

We’ve learned to rehearse, to “dry run” all significant operations, and to challenge each other to approach the theoretical limits of operational goodness – to have zero downtime even for significant system changes.

All this requires very different thinking from traditional enterprise systems. It is even different from running traditional high-volume websites, because we are responsible for very different websites across far flung, geographically diverse domains, each with highly isolated data stores running on a shared fabric.

We love the enormous amounts of learning this involves, the need to constantly improve ourselves and our systems to handle constantly increasing loads. If you’ve built a high-scale SaaS system and evolved it over time into a best-of-breed solution that is truly stable under enormous load spikes. If you’ve learned to be a real engineer under these conditions – to understand how to solve problems thoroughly and deeply and to consider issues of monitoring, scale, failure, analytics, concurrency, availability and consistency, then there are few systems you cannot be confident that you cannot build.

On the Powers and Dangers of Caching

Operating systems, the internet, video games, social media – they all would not exist without significant hardware and software infrastructure devoted to caching. The same is true for us: We make enormous demands on a host of distributed caching solutions, some open source, some commercial, all carefully optimized for a particular workload.

Our front line of defense is our content distribution system, which replicates content that can be statically cached, out to network edges for speedy delivery. Automated tasks in our system determine which pieces of content are globally usable and push them out to the edges. Our JavaScript infrastructure does the same on browsers, pulling in data from the correct places at the right times so as to get optimal performance. This allows us to handle enormous traffic spikes.

A second line of defense is our web-tier content cache, which serves content to authenticated/connected users. This is a large, but all-in-all straightforward caching system.

More interesting is our query result cache system, a multi-level distributed caching system that lazily replicates cache data from centralized memcached servers to in-memory caches on our web tiers. This enables us to handle cache system failures gracefully, so that cache restarts do not result in massive load spikes on our database clusters. Indeed, this is one of the main dangers of caching: When caching servers go down, rewarming those caches can drive back-breaking load on back-end systems, and so managing cache hydration and persistence is a key task for systems with 24×7 uptime.
Also interesting is our approach to cache locality: Since modern databases can serve single-row queries on clustered-index data incredibly quickly, improving object fetch performance is best done by avoiding network round trips, rather than query overhead. Therefore, we store query results in remote caches, but never objects, which, if they are cached at all, are always cached on the web tiers.

Using these approaches, as well as many others, such as sophisticated ETag exchanges, we enable our infrastructure to scale to the tens of thousands of requests per second that our customer base drives at peak hours.