Designing Data-Intensive Applications

The argument

Kleppmann opens by dissolving a category. By the end of the first chapter there is no such thing as the database, singular, as a kind of object: there are storage engines, caches, indexes, message queues, and stream processors, and the modern application is a composite of them, stitched together by application code that has quietly become a distributed data system of its own. The developer who believed he was choosing a database was in fact designing one, without the vocabulary and without the map. The book exists to supply both, and its method is to hold every mechanism up against three questions. Reliability: does the system continue to work correctly when hardware faults, software bugs, and human error arrive, as they will. Scalability: is there a described strategy for growth, with load parameters named, rather than a hope. Maintainability: can the thing be operated, understood, and changed by people who were not in the room when it was built. Everything that follows, eleven chapters of machinery, is these three questions asked at descending depths.

The first descent is into data models, and it is quietly the most opinionated part of the book. A data model is an instrument of thought: the relational model, the document model, and the graph models each make certain questions cheap and others nearly unaskable, and the choice determines what the organization will find easy to know about itself. Kleppmann’s history here has a point to it. The document databases of the 2010s replay the hierarchical model of the 1960s, and the case against them was already argued when IMS ran on mainframes; the network model lost to relations once before, for reasons that did not expire. He is not sneering at either side. He is observing that the industry’s debates are generational, and that an engineer who knows the previous round can price the current one correctly.

Below the models sit the storage engines, and here the book performs its most useful simplification: two families. The B-tree updates pages in place, keeps data sorted, and has earned four decades of trust; the log-structured merge tree appends, sorts, and compacts in the background. Each buys its performance from a different account, write amplification on one side, read amplification and compaction debt on the other, and the correct choice is a property of the workload, not of the era or the vendor. The section on encoding closes the foundations with the least glamorous and most organizational material in the book: schemas, and the discipline of forward and backward compatibility, are what permit rolling upgrades, which is what permits the system to change while running, which is what permits the organization to change at all. Avro against Thrift against protocol buffers looks like a dull decision. It is a decision about whether next year is possible.

Then the book distributes the data, and the tone hardens. Replication comes in three topologies. Single-leader is the sane default and introduces replication lag, whose anomalies have names users experience without knowing them: reading your own write and not finding it, watching time move backwards between two reads. Multi-leader buys write availability at the price of conflicts that must be resolved by somebody, usually badly. Leaderless systems in the Dynamo style push the problem into quorums, read repair, and the discovery that overlapping majorities guarantee less than intuition claims. Partitioning gets the same treatment: hash partitioning trades away range queries, range partitioning invites hot spots, rebalancing and secondary indexes each carry costs that appear only at the worst time. The transactions chapter should be mandatory for anyone who has ever said the word ACID in a sales meeting. Isolation levels are weaker than their names; the levels most applications actually run permit write skew and phantoms; true serializability must be bought with one of three specific mechanisms, serial execution, two-phase locking, or serializable snapshot isolation, each with a bill attached.

Underneath all of it lie the two chapters that justify the book: the trouble with distributed systems, and consistency and consensus. Networks lose and delay packets without limit; clocks drift and jump despite NTP; processes pause for garbage collection at any moment and for any duration. From inside the system, a slow node and a dead node are the same object, so truth becomes what a majority of nodes agree it is, and safety requires mechanical humility: fencing tokens, idempotence, designs that assume partial failure as the normal weather. On this floor Kleppmann does the field a service by demoting the CAP theorem, which he treats as too narrow to guide real designs, and promoting the real hierarchy: linearizability, total order broadcast, and consensus are equivalent problems, all expensive, all concentrated into tools like ZooKeeper precisely so that ordinary systems can rent consensus rather than reimplement it.

The final part reorganizes the whole book around one distinction: systems of record and derived data. Indexes, caches, materialized views, search indexes, recommendations: all derived, all rebuildable in principle from a source of truth. Batch processing, told through MapReduce as the Unix philosophy at datacenter scale, and stream processing, told through log-based brokers and change data capture, turn out to be one computation at two latencies. The closing chapter proposes the consequence: unbundle the database, let an ordered log of events be the spine, and let stream processors maintain every derived view, with correctness enforced end to end at the application boundary rather than trusted to any single component. Then, in a coda that most technical books would not risk, he turns to ethics: data collected without purpose becomes surveillance by default, an industrial waste product with a long half-life, and the engineer who builds the pipeline owns a share of what flows through it.

Working notes

The book is a map, and the recurring mistake of its readers is to use it as a menu. A map tells you what the terrain costs; a menu invites you to order. Every chapter of this book prices some piece of distributed machinery, and the price is the point: the correct reading of most chapters is relief that you do not need the thing described. It is fitting that each chapter opens with a drawn map, terrain and hazards sketched like a medieval chart. Kleppmann is telling you, in the endpapers, what kind of book he wrote.

Its second gift is vocabulary, which is armor. An architect who can say write skew, monotonic reads, sloppy quorum, and fencing token cannot be sold to as easily as one who can only say fast and scalable. Most of the data-platform failures I am hired to repair began in a meeting where one side owned the vocabulary and the other side owned the budget. The book transfers the vocabulary to the side that pays, which is a genuine redistribution of power, and I suspect it has quietly saved more money than any framework of its decade.

Read against Brooks, the book slots neatly into the essence and accident distinction. Vendors sell the removal of accident: faster engines, managed clusters, friendlier consoles. Kleppmann catalogues essence: the anomalies, the impossibilities, the trade-offs that survive every product cycle because they are properties of physics and arithmetic, not of implementations. Replication lag does not care which decade you deploy it in. A book organized around essence does not age the way its examples do, which is why a snapshot of 2017 still reads as current while the 2017 marketing it displaced reads as archaeology.

The derived-data distinction is the piece I now use daily, and it connects to the systems literature more than to the database literature. One stock, many flows: a single system of record, and every other representation a materialized consequence that can be dropped and rebuilt. Meadows would recognize the shape immediately. Most enterprises run the inverse pattern, a dozen systems each holding a fraction of the truth, reconciled by batch jobs and apology, and the nightly reconciliation meeting is the interest payment on an architectural debt nobody wrote down.

What only became visible on second reading is the book’s honesty about restraint. The sentences recommending a single machine, the warnings that distributed transactions will halve your throughput, the repeated observation that the simplest topology that meets the requirement is the correct one: these are the load-bearing sentences, and they are the ones the conference-talk culture filtered out. The book earned its reputation as the distributed-systems bible; its actual doctrine, read carefully, is closer to: do not distribute until you must, and know exactly what you surrender when you do.

Where I push back

The book’s completeness produces a specific illusion in its readers: knowing the taxonomy of trade-offs feels like having made them. It is not. Kleppmann can tell you what snapshot isolation permits and forbids; he cannot tell you whether your settlement process can tolerate write skew, because that is a question about your business, your regulator, and your appetite for a specific loss, and no amount of mechanism answers it. The engineer who finishes this book has a map and no destination, and the industry is full of well-mapped systems built toward destinations nobody priced. Taxonomy is not judgment, and the book’s authority makes the substitution easy to miss.

The closing vision overreaches. Unbundling the database, the log as spine, every view a stream-maintained derivation: it is intellectually clean and operationally aristocratic. It assumes an engineering organization that can run brokers, processors, and schema governance as a permanent discipline, which is to say it universalizes the staffing of the companies where the pattern was born. The bundled database that Kleppmann politely deconstructs is, for most organizations, the correct amount of bundling: transactions, constraints, and recovery in one artifact, operated by two people instead of twelve. His future is real, and it arrives selectively, and the chapter does not say loudly enough that for most readers it should not arrive at all.

And the book is silent about the layer where enterprise data actually fails: the political one. Ownership, stewardship, the department that will not share its tables, the metric defined three ways by three vice presidents. Kleppmann solves the problems that have solutions in mathematics. The ethics coda, decent and heartfelt, treats an incentive problem as a matter of engineering conscience; pollution was not ended by asking factories to feel responsible, and data hoarding will not be either. The hardest problems under every data platform I have rebuilt were jurisdictional, and on those this map is blank.

How it enters the work

Intelliblitz builds data platforms for enterprises, and this book supplies the doctrine under all of them: one system of record, named in the contract, and everything else derived and rebuildable. When a client can drop any dashboard, index, or feature store and regenerate it from the source of truth, the platform is honest; when they cannot, we have found the real project. The client owns the log, not rents it, which aligns the book’s architecture with the way I insist engagements end: systems the client operates without us.

The vocabulary runs procurement. Before any vendor meeting, we write the guarantee sheet: which anomalies the business can survive, which isolation level the workload actually requires, what the recovery point objective is in minutes rather than adjectives. Vendors answer a sheet like that or they leave; the difference between fast and linearizable has, more than once, been the difference between a defensible architecture and an expensive apology. Most of what a client pays enterprise consulting for is contained in knowing which questions from chapters seven and nine to ask before signing.

The defaults audit is standing practice because the book taught me where defaults hide. The BI report that reads a lagging replica and contradicts the transactional system by a few minutes is not a bug in the report; it is a replication topology surfacing in a boardroom, and someone who has read the anomalies chapter recognizes it in one sitting rather than one quarter.

Above all the book disciplined my restraint. The most valuable recommendation I make, several times a year, is that a client needs one well-run relational database and no distributed anything: fewer moving parts, one truth, boring on purpose. It takes authority to sell simplicity to a committee that arrived wanting a platform, and I borrow that authority partly from this book, which prices the complexity they were about to order. Find the structure, then remove everything that is not it: Kleppmann wrote six hundred pages of what the removal costs when you skip it, and I bill for the removal.