The Future of Data Systems: The Unbundled Database, Kappa, and End-to-End Correctness (Finale)

· tech

#distributed-systems#book-notes#data-engineering

📑 Contents

The final chapter. The first eleven took storage, replication, partitioning, transactions, consensus, batch and streams apart piece by piece; here Kleppmann gathers them into one bold perspective: stop treating “the database” as a box — take it apart. And by the time you get here, you’ll realise you already live inside that “future”.

Unbundling: your data platform is a database taken apart

A database is really a bundle of features: storage engine, indexes, cache, materialized views, replication log — all in one box, kept consistent by the DB’s internals. Kleppmann’s observation is that modern data platforms are taking that box apart — each feature handled by a specialised system, and the glue keeping them consistent is exactly the log from the previous post:

Traditional: one box, everything bundled storage engine indexes cache mat. view log ↓ unbundle: each feature handed to a specialised system ↓ log as hub (Kafka) — the glue that sets order OLTP DBstorage + transactions Elasticsearch= the index Redis= the cache warehouse / Gold layer= materialized view every system is a follower of the log, consuming in the same order → each consistent Your data platform = a database "turned inside out"; what holds it together is the log
The top half is a traditional database: storage, indexes, cache and materialized views bundled in one box, kept consistent by the DB's internals. The bottom half is unbundling: the same features each claimed by a specialised system — the OLTP DB handles storage and transactions, Elasticsearch is the index taken out, Redis is the cache taken out, the warehouse is the materialized view taken out — and what keeps them consistent is the log in the middle (every system is its follower). In other words: the data platform you run is, at heart, a database turned inside out — and the discipline for designing it should be the same as a database's: the log sets the order, derived data can be rebuilt

The practical value of this perspective: “add search to the platform” = “build an index on this big database” — and the method is the same: replay from the log, grow a new follower, touch nothing existing. When the index is built into the database, you trust it to maintain itself; once unbundled, the responsibility for keeping “the index caught up with the primary data” lands on you — and that’s the deep reason the data engineer profession exists.

Lambda vs Kappa: two sets of logic, or one log

For “the same data has to be computed both accurately (batch) and fast (real time)”, history offers two answers:

Lambda: two tracks data arrives batch layerperiodic full recompute (accurate) speed layerstream fills in real time (fast) query: "merge" both results ✗ same logic written twice (batch + stream), 2× upkeep Kappa: one log one replayable log (kept long enough) stream job v1always computing live stream job v2recompute: replay from start switch once caught up → one codebase, recompute = replay requires: log kept long enough, replay throughput sufficient Logic changes often, can't afford two stacks → kappa; recompute too big for stream replay → keep the batch track
Lambda: a batch layer periodically recomputes everything (accurate) + a speed layer streams to fill in real time (fast), and a query merges both — the price is that the same business logic is written twice in two frameworks, double the maintenance, and debugging hell when the two disagree. Kappa: keep only one replayable log and one set of streaming logic; to recompute history (changed logic, fixed bug), start a new job replaying from the beginning of the log and switch over once it catches up — batch's "human fault tolerance" achieved in streaming form. The precondition is a log kept long enough and replay throughput that holds up; unified batch/stream engines (one program, two modes) are also removing the "write it twice" pain at the framework level

The most honest lesson: the last mile of correctness is end to end

In the book’s final technical section, Kleppmann throws an important bucket of cold water: don’t trust any middleware’s guarantee blindly. Exactly-once is powerful, but its guarantee has a boundary — once data leaves it (written to an external system, an external API called, a user pressing submit twice), the semantics break. The only truly reliable deduplication is end to end: the request carries a unique request id (idempotency key) from its source all the way to the final write, and the endpoint does the final check — a replay in data systems of networking’s ancient end-to-end argument, and the theoretical basis for the discipline of idempotency. One level up, he argues systems should audit: periodically verify data integrity (do the counts match, do the sums match) rather than assuming “the pipeline didn’t error = the data is right” — no error only means no error was found. That sentence is the starting point of data-quality engineering.

Reflections

”The platform is a database taken apart” — this perspective flipped my whole map over

Unbundling is the idea with the biggest recoil after finishing the book. Looking back at the platform I run myself: Kafka is the commit log, Elasticsearch is the index, Redis is the cache, the warehouse’s Gold layer is the materialized view — what I operate every day is really a giant database spread out across K8s, and my job is the job of a database kernel engineer: keep these “unbundled components” consistent with the primary data. This perspective immediately gave me two disciplines. In design, any new component should be a follower of the log, not another victim of dual writes; in debt, every bit of the consistency a database guarantees “for free” through transactions has to be repaid by hand once unbundled — so before taking it apart, ask: can a single-machine database really not do this any more? That question again.

”No error ≠ the data is right” — auditing is data engineering’s next stop

The end-to-end lesson is a tailor-made reminder for my role. After years of SRE it’s easy to treat “the monitoring is green” as “everything’s fine”; Kleppmann punctures it: an all-green pipeline only means the system didn’t error, not that the data is right — row counts quietly 2% short, a join silently matching nothing, an amount column in the wrong unit, and monitoring says not a word. A system’s reliability relies on monitoring; data’s correctness relies on auditing — the latter is independent verification (reconciliation, count checks, invariant checks), not reading logs. That’s exactly the subject of the Data Quality series I want to start next: turning “is the data right” from a prayer into an engineering discipline with metrics, alerts and SLOs. DDIA’s last chapter happens to be the best introduction to that series.

Closing the book: it gives not answers but a framework of questions

Twelve chapters in, if you ask me what DDIA actually gave me, my answer is: a list of questions that will never go out of date. How is the data laid out (the shape of the reads)? Where does state live, who is the source of truth? Who decides the order, what protects it? Where is the boundary of the guarantee, who checks the last mile? — tools will change (plenty of the book’s examples are already old), but these questions you’ll ask until you retire. It’s also where this whole blog converges: DDIA is the principles layer, Redis/Kafka/Spark/K8s the implementation layer, SRE/LGTM the operations layer — one web: principles let you understand the tools, tools let you verify the principles, operations makes you pay the price for both and earns you the respect. Finally, Kleppmann chooses to end with a full section on ethics: data is power, and power needs restraint — in this profession we decide daily “what to record, how long to keep it, who gets to see it”, and that has never been only a technical decision. The more capable the system, the more restrained the engineer needs to be — and that’s where this book ends.