Reconciliation: We Never Built It, So Why Did the Books Balance?

In the plan this chapter was called “the three ledgers: stock, orders, payments”, and it was going to be about how we reconciled them. As usual I went back to check what we actually did before writing — and the answer is: we didn’t reconcile. There was no reconciliation job in the system, no balancing at the end of a round, and after launch we barely ever handled a wrong figure.

A system handling real money, with no reconciliation mechanism, whose books were almost never wrong — that sentence deserves more explanation than any reconciliation architecture. The answer is scattered across the previous fifteen chapters, and this one gathers it up.

An unbundled database

Start with a realisation that only surfaced halfway through the series. One day I was staring at the whole system and it suddenly looked familiar: everything we built is a scaled-up version of a database’s internals.

Landing comments after cleaning is the WAL (#3); the FSM batch digesting comments into carts is a log consumer building indexes; the sold quantity is a materialised view (#5); payment status derived on read is a view (#7); the allocation log being rebuildable end to end is a redo log (#8); heartbeat table scans are the built-in scheduler (#15); and the hourly recomputation is repair. DDIA’s final chapter calls this an unbundled database — take a database apart and reassemble it out of individual components. We’d never read that chapter, and spent a year and a half building it.

Unbundling isn’t free. Inside a database an index always keeps up with the heap, a materialised view has refresh guarantees, and a transaction covers everything; unbundled, those guarantees are yours to restore. Reconciliation is in theory the last line of “restoring consistency yourself” — so “we didn’t reconcile” needs an accounting.

Three ledgers, and which of them drifts

Lay out the three ledgers and their sources of truth:

• The stock ledger: cap plus sold quantity, one dedicated table, two columns.
• The order ledger: order and order item, with amounts frozen into accounting facts at the moment of sale.
• The payment ledger: orders payment, backed by per-provider payment fact tables.

For a ledger to go wrong, the necessary condition is the same fact existing in two places, each updated separately — only redundancy drifts. Measure the three ledgers with that ruler and it gets interesting:

The order and payment ledgers carry almost no redundancy. An order’s “status” isn’t a column, it’s derived from facts at read time; payment progress isn’t a boolean, it’s the sum of per-provider fact tables. Promotion amounts use floor-and-subtract, where the totals matching is guaranteed by the algorithm rather than checked afterwards. With no second ledger that can drift, there’s no ledger to reconcile — that isn’t us reconciling well, it’s these two ledgers structurally abolishing the need.

The stock ledger has exactly one redundancy. The sold quantity is the one number in the whole system deliberately materialised — for the speed of a live moment, there was no alternative. And it really did drift: the oversell after that migration is exactly this number knocked out of true by a requirement change. Its line of defence is the hourly recomputation: recompute the count wholesale from the facts in carts and orders. One redundancy, one repair loop, balanced.

So the first layer of the answer to “no reconciliation”: the need for reconciliation is proportional to materialised redundancy. We squeezed redundancy down to one, and reconciliation shrank to one scheduled job — so quiet that nobody ever called it reconciliation.

The real reconciliation happens outside the system

The second layer: reconciliation did exist, just not in the system — it was in the accounting department.

The system’s responsibility ends at the export: organising orders the way accounting wants them. That’s really a translation layer — turning an engineer’s fact tables into accounting’s language. Execution, judgment and correction all happen in the human world: accounting finds a mismatch and asks the CTO to assign an engineer to investigate; once investigated, the fix depends on the direction — undercharged, an engineer adjusts the DB so the amounts line up and the company absorbs it internally; overcharged, support contacts the customer and refunds loyalty points or cash as external compensation.

That asymmetry is worth a look. Undercharging is the company’s loss with no effect on the customer, so it’s handled quietly and internally; overcharging touched a customer’s money, so it goes through the most expensive channel (a support call) with the most sincere compensation. The direction of the correction decides who pays and in what currency — mechanism to the system, policy to people, again: the system supplies facts and people decide justice.

And the actual record of all that: after launch, we barely remember handling a wrong figure. Even closing a round, a settlement touching thousands of rows at once, only did blocklisting, cart clearing and unpaid-order clearing — no balancing, and it ran surprisingly smoothly regardless. At the time we credited that to obediently following 3NF; now I can put it more precisely: 3NF is the discipline of “one fact stored once”, and it strangles drift at the source.

The rebuild: turning “almost never wrong” into “provably right”

So would a rebuild still do nothing about reconciliation? No. Between “the books were almost never wrong” and “the books can be proven right” lies the same gap #15 described: our peace of mind was “we got used to nothing going wrong”, not “the numbers say nothing is wrong”. A rebuild adds three small things, all growing along the existing structure:

1. Self-reconciliation queries, on a schedule. The invariants are already writable: each order’s amount equals the sum of its order items, each orders payment’s receipts are at least the orders it covers, sold quantity equals the sum of reserving carts and orders. Run them daily on a schedule, silent when green and alerting when red — a few lines of SQL bought in exchange for turning “should be fine” into “verified fine”.
2. Fix the books with a compensating entry, not by adjusting the DB. Back then undercharging was fixed by adjusting the DB to match — understandable, but strictly speaking it’s rewriting history: six months later nobody remembers why that number looks the way it does. Accounting’s own rule is the right one: don’t erase an error, add a reversing entry. The history of the books is a fact too, and this series’ iron law — append facts, derive status — shouldn’t get an exemption at correction time.
3. Manage the export format as a contract. Accounting’s requirements change, and the export is the only interface between the system and them — version it, keep samples, review format changes, and treat it like an API.

Just those three. No reconciliation platform, no daily full comparison — in a system with exactly one redundancy, building a bigger reconciliation system is fighting an enemy that doesn’t exist.

Reflections

The best reconciliation is leaving the books no chance to be wrong. The need for reconciliation is proportional to materialised redundancy — each extra copy of a derived number is another ledger that can drift and another checking job to write. Leaving derived data on the read side is the cheapest consistency there is: it makes the word “reconciliation” nearly disappear from the system’s vocabulary. We weren’t good at reconciling; we designed away the reasons to reconcile, one at a time.

“Nothing went wrong” isn’t “provably nothing is wrong”. This is #15’s credit argument replayed on data: our books were clean, but no measurement could prove they were clean — the same silence as the risk chapter‘s “no complaints received”. The design dividend was real, but a dividend needs a report card, and three invariant queries buy exactly that.

The deepest lesson this system taught me is this chapter’s title read backwards. “We never built X, so why did nothing go wrong?” — substitute reconciliation, the state machine, the notification abstraction layer, and the previous fifteen chapters keep showing the same structure: not building it, because the structure made it unnecessary. Engineering maturity isn’t ticking off everything on the list, it’s knowing which items your structure already gave you free and which you must restore by hand. On the unbundled database laid out across the table, the piece restored most successfully is precisely the one where it looks like nothing was done at all.

Next chapter, something smaller and prettier: images. Uploading takes an afternoon; deleting takes a lifetime — the life and death of a resource.