Stream Processing: The Dual-Write Trap, CDC, and Stream–Table Duality

· tech

#distributed-systems#book-notes#streaming

📑 Contents

Batch processes data that has “all arrived”; streaming processes data that “keeps coming”. Much of the groundwork for this chapter I’ve laid elsewhere: logs and offsets in the Kafka series, delivery guarantees in the delivery post, windows and event time in Spark Streaming — none of it repeated here. DDIA Ch11’s real specialty is three more fundamental ideas: why “writing two copies separately” is doomed, how to make the database itself the source of events (CDC), and how “stream” and “table” are two faces of the same thing.

The dual-write trap: the same data, written separately to three systems

In real systems the same data often has to live in several places at once: the DB as primary store, Redis as cache, Elasticsearch as search index. The intuitive approach is for the application to write all three itself (dual write) — and that’s the source of the most common silent data incident:

✗ Dual write: the app writes three copies application DB cache search index disease 1: crash halfway → some written, some not no transaction can roll back across three systems disease 2: concurrent writes arrive in different orders DB gets A then B, cache gets B then A → different values → three systems diverge permanently, silently ✓ Log first: write to one place only application only writes here one ordered log (source of truth) DB cache search index all consume in the "same order" → consistent, replayable every downstream is a follower, converging to one state One dataset into N systems? Pick one source of truth, make all the rest followers
Dual write's two diseases have no cure: partial failure (the app crashes after writing the DB, the cache never catches up — no transaction can roll back across systems) and reordering (two concurrent writes arrive at the three systems in different orders, each converging to a different value) — the three systems diverge permanently, and silently. Log first eliminates the problem structurally: write to one place only (one ordered log), and every downstream consumes it in the same order — consistent ordering, and anything lost is replayed from an offset. This is really the leader–follower of the replication chapter, generalised across heterogeneous systems: pick one source of truth, and make everything else a follower

CDC: making the database itself the source of events

“Log first” sounds like rewriting the whole application — but there’s a clever shortcut: the database already has an ordered log of writes (the WAL / binlog, which is what replication followers sync from). CDC (change data capture) takes that internal replication log and turns it into an event stream anyone can subscribe to — tools like Debezium pose as a replication follower and write every change into Kafka. The application changes not a line and keeps writing the DB as before; cache, index and warehouse all switch to consuming the stream. The DB remains the source of truth, but every one of its heartbeats is heard by the whole world — and that’s the mainstream way modern data platforms feed OLTP data into the analytics side.

Stream–table duality: a table is the integral of a stream, a stream is the derivative of a table

The most beautiful idea in the chapter is that a stream and a table are two faces of the same thing:

Stream: a changelog ① k1=a ② k2=x ③ k1=b (overwrites a) ④ k2=∅ (delete) every "change" is an event, in order fold up to now (apply each) emit one per change (changelog) Table: the state right now k1 = b (k2 deleted) the same information, frozen at "now" log compaction = keep only the last entry per key (smallest stream that rebuilds the table) · materialized view = a table that keeps folding
A stream is the sequence of "every change"; fold it (apply in order) from start to end and you get the table — the state right now. Conversely, emit every change to the table as an event and you recover the stream. The engineer's phrasing: a table is the integral of a stream, a stream is the derivative of a table. A pile of things you've seen are its incarnations: log compaction (keep only the last entry per key = the smallest stream that rebuilds the table), materialized views (a table that keeps folding), replication streams (turn the leader's table back into a stream, send it to the follower, fold it back into a table). State machine replication is it too: the log is the stream, each node's state is the table

The practical consequence of this equivalence: you can keep the “stream” forever and treat the “table” as a derivative that can be thrown away and rebuilt at any time. Cache broken? Fold it again from the log. Want a new search index? Replay the log from the start and a new follower grows. The batch chapter’s “human fault tolerance” — immutable inputs, rerun when wrong — is carried intact into the streaming world by the log: as long as the log is there, all state is just a cache.

Reflections

”Who is the source of truth?” — one question that stumps nine out of ten data architectures

That dual-write diagram is the incident archetype I’ve seen most often at work: DB and cache disagree, the ES index doesn’t match the primary, warehouse numbers are off from production — trace it to the root, and nearly always somewhere is writing two copies separately, and nobody is anybody’s follower. So now, looking at any data architecture, my first question is always: who is the source of truth for this data? Do the other copies “follow the same ordered log”, or “each write their own and pray they agree”? If it’s the latter, it just hasn’t broken yet. And CDC is elegant precisely because it doesn’t ask you to rewrite the application — it promotes the database’s existing replication mechanism from internal plumbing to a public interface, turning “add a follower” from a big project into subscribing to a stream.

”A table is the integral of a stream” — the third equivalence that unlocks a whole class

That’s the third “one equivalence, a whole class understood” I’ve collected in this series: consensus = one log everyone agrees on, batch = a pure function over immutable input, and now table = the fold of a stream. In one stroke it strings scattered things together: why a Kafka compacted topic can back a KTable, why a materialized view is called “materialized” (freezing a stream into a table), why Redis replication sends a stream of commands instead of the whole dataset, why Kafka Streams dares to keep its state store local (the changelog is in Kafka, fold it again if it’s lost). Even Medallion can be retold in its terms: Bronze is the archived stream, Silver and Gold are folds of different depths. An abstract equivalence is the highest compression ratio knowledge has.

”As long as the log is there, all state is just a cache” — that sentence is worth an architecture

Having written this post, I want to pull out the single most powerful sentence in the series: treat the immutable log as the only truth, and all state (cache, index, report, even the database itself) as a rebuildable derivative. It reduces the hardest question, “what do we do when it breaks”, to “replay the log once”; it reduces “we want a new view” from a migration project to “start a new consumer from the beginning”. That’s the final payoff of the log vs queue axis — choosing “keep it” over “take it” back then is what earns the right to say “everything can be rebuilt” today. It isn’t free, of course: how long to retain the log, a schema that can evolve, replays that are idempotent — all taxes to pay. But as an architecture’s default leaning, I’m now entirely on the log’s side. The next post is the book’s final chapter: assembling these pieces into Kleppmann’s vision of the future of data systems.