Batch Processing: The Spirit of MapReduce, Two Roads to a Join, and the Virtue of Immutable Inputs

Part II held consistency together inside a single system; Part III’s theme switches to data flowing between systems — and the oldest, most reliable way for it to flow is batch. DDIA’s way into MapReduce is distinctive: it starts with the Unix philosophy — small tools, each doing one thing well, connected by pipes through a uniform interface (files and streams). Then it names the theme in one line: MapReduce is a Unix pipe across a thousand machines — immutable inputs, output written as new files, tools linked by a uniform format. That spirit has outlived MapReduce itself by a long way.

Anatomy of MapReduce: map, shuffle, reduce — the expensive one is in the middle

Take “count clicks per URL” as the example; three steps and done:

Two roads to a batch join: move everything, or bring a cheat sheet

The most common heavy lifting in the batch world is the join (click logs joined with a user table). Single-machine join algorithms I covered in the SQL series; the core question of the distributed version becomes: two datasets are scattered across different machines — how do rows with the same key meet? Two roads:

MapReduce itself has a big drawback: every job’s output has to be fully materialised to HDFS and read back by the next job — a ten-step pipeline writes and reads disk ten times. The later dataflow engines (Spark, Flink) evolved precisely against this: draw the whole pipeline as a DAG of operators, keep intermediate results in memory as far as possible, and recover from failure by recomputing locally from lineage. MapReduce the “product” was replaced, but its spirit — partitioned parallelism, bringing compute to the data, concentrating cost in the shuffle — lives on unchanged in every modern engine.

Batch’s most underrated virtue: inputs are immutable, so mistakes can be redone

There’s an observation in this chapter that’s easy to skip and that I consider the most important: batch processing inherits Unix’s best character trait — inputs are read-only, outputs are written somewhere new. That gives a capability the author calls “human fault tolerance”: the code was wrong, the logic had a bug, yesterday’s report came out broken — fix the code and rerun against the untouched input. Errors don’t accumulate, don’t pollute the source; the worst case is a wasted round of computation. Compare a pipeline that UPDATEs the database directly: one bug corrupts the only truth, and recovery takes backups and tears. Rerunnability is the greatest gift batch gave data engineering — Medallion keeping an immutable Bronze layer, Airflow’s idempotency and backfill, are all modern incarnations of this virtue.

Reflections

”Human fault tolerance” is the line I most want to say on batch’s behalf

People call batch old and slow, but after years in data what I’m most grateful to it for is precisely this unsexy virtue: people will make mistakes, and batch makes mistakes reversible. Logic wrong? Fix and rerun. Yesterday’s dimension table broken? Rebuild from Bronze. That sense of safety — “if it’s wrong, you can go back to the start” — costs ten times the effort to obtain in the streaming world (once an event has flowed past, it’s gone). So the discipline I give teams has always been: the raw layer is read-only and kept forever, and every downstream is treated as a derivative that can be “burned down and rebuilt at any time” — that’s the soul of Medallion, and its theoretical basis is in this chapter. System fault tolerance relies on replicas; human fault tolerance relies on immutability — the latter is discussed far too little and causes far more incidents.

Shuffle is the rent of distributed computing — understand it and tuning has a map

Of MapReduce’s three steps, map and reduce both run “in place”; only the shuffle moves data — and nearly the entire cost of distributed computing lives in that step. Once that clicks, a pile of scattered practical knowledge snaps into place: why Spark‘s stages are cut at shuffle boundaries, why data skew is deadly (all of one key’s data crammed onto one node), why broadcast join is fast (it skips the shuffle entirely), why filtering before a groupBy saves so much (shrink before you move). Now, faced with any slow batch job, my first question is always: how much data does it shuffle? Can it move less, shrink earlier, or just bring a cheat sheet and not move at all? That one question is half of tuning.

The tool died, the philosophy lives — learn things down to that layer

MapReduce as a product has been retired, but the chapter reads as anything but dated, because it teaches three ideas that are still alive: bring the computation to the data (not the reverse), use immutable files as the uniform interface between tools (the Unix pipe writ large), and concentrate cross-machine complexity in one place, the shuffle. Spark is their new shell; dbt’s model chains and Medallion‘s layering are, underneath, the same “each step reads an immutable input and writes a new output”. It confirms once more the order in which I learn technology: APIs change every year or two, architectures every three to five, but design ideas at this level serve you for twenty. That’s the point of reading a book like DDIA — when the next new tool ships, you recognise at a glance “oh, that idea again in new clothes”. The next post covers its other half: when data no longer arrives “a batch at a time” but “all the time” — streams.