Reconciliation: We Never Built It, So Why Were the Books Still Right?

In the plan this chapter was called “the three ledgers: stock, orders, payments”, and it was going to be about how we reconciled them. As usual I went back to check what we actually did before writing, and the answer is: we didn’t reconcile. There was no reconciliation job in the system, no balancing at the end of a round, and after launch we barely ever handled a wrong figure.

A system handling real money, with no reconciliation mechanism, whose books were almost never wrong — that sentence deserves more explanation than any reconciliation architecture. The answer is scattered across the previous fifteen chapters, and this one gathers it up.

An unbundled database

Let me start with a realisation that only surfaced halfway through the series. One day I was staring at the whole system and it suddenly looked familiar: everything we built is a scaled-up version of a database’s internals.

Landing comments after cleaning is the WAL (#3); the FSM batch digesting comments into carts is a log consumer building indexes; the sold quantity is a materialised view (#5); payment status derived on read is a view (#7); the allocation log being rebuildable end to end is a redo log (#8); heartbeat table scans are the built-in scheduler (#15); and the hourly recomputation is repair. DDIA’s final chapter calls this an unbundled database — take a database apart and reassemble it out of individual components. We’d never read that chapter, yet we spent a year and a half building it.

Unbundling isn’t free. Inside a database an index always keeps up with the heap, a materialised view has refresh guarantees, and a transaction covers everything; unbundled, those guarantees are yours to restore. Reconciliation is, in theory, the last line of defence in “restoring consistency yourself” — so “we didn’t reconcile” needs an accounting.

Three ledgers, and which of them drifts

First, the three ledgers and their sources of truth, laid out:

• The stock ledger: a stock cap plus two consumption counts (cart / order), one dedicated table, three numbers.
• The order ledger: order and order item, with amounts frozen into accounting facts at the moment of sale.
• The payment ledger: orders payment, backed by per-provider payment fact tables.

For a ledger to go wrong, the necessary condition is the same fact existing in two places, each updated separately — only redundancy drifts. Measure the three ledgers with that ruler and it gets interesting:

The order and payment ledgers carry almost no redundancy. An order’s “status” has no single aggregate status column — each dimension follows its own facts directly, and the aggregate reading happens at read time; payment progress isn’t a boolean, it’s the sum of per-provider fact tables. Promotion amounts use floor-and-subtract, so the sum identity is guaranteed by the algorithm rather than checked afterwards. With no second ledger that can drift, there’s no ledger to reconcile — that isn’t us reconciling well, it’s these two ledgers structurally abolishing the need.

The stock ledger has exactly one redundancy. The sold quantity is the one number in the whole system deliberately materialised — for speed mid-stream, there was no way around it. And it really did drift: the migration oversell is exactly this number knocked out of true by a requirement change. Its line of defence is the hourly recomputation: recompute the count wholesale from the facts in carts and orders. One redundancy, one repair loop, and the books balance.

So the first layer of the answer to “no reconciliation”: the need for reconciliation is proportional to materialised redundancy. We squeezed redundancy down to one, and reconciliation shrank to one scheduled job — so quiet that nobody ever called it reconciliation.

The real reconciliation happens outside the system

The second layer: reconciliation did exist, just not in the system — it was in the accounting department.

The system’s responsibility ends at the export: organising orders the way accounting wants them. That’s really a translation layer — turning an engineer’s fact tables into accounting’s language. Execution, judgment and correction all happen in the human world: accounting finds a mismatch and asks the CTO to assign an engineer to investigate; once investigated, the fix depends on the direction — undercharged, an engineer adjusts the DB so the amounts line up and the company absorbs it internally; overcharged, support contacts the customer and refunds loyalty points or cash as external compensation.

That asymmetry is worth a look. Undercharging is the company’s loss with no effect on the customer, so it’s handled quietly and internally; overcharging touches a customer’s money, so it goes through the most expensive channel (a support call) with the most sincere compensation. The direction of the correction decides who pays and in what currency — mechanism to the system, policy to people, again: the system supplies facts and people decide justice.

And the track record of all that: after launch, we barely remember handling a wrong figure. Even closing the round, that big settlement touching thousands of rows at once, only did blocklisting, cart clearing and unpaid-order clearing — no balancing, and it ran surprisingly smoothly regardless. At the time we credited that to obediently following 3NF; now I can put it more precisely: 3NF is the discipline of “one fact stored once”, and it strangles drift at the source.

The rebuild: turning “almost never wrong” into “provably right”

So would a rebuild still do nothing about reconciliation? No. Between “the books were almost never wrong” and “the books can be proven right” lies the same gap #15 described: our peace of mind was “we got used to nothing going wrong”, not “the numbers say nothing is wrong”. A rebuild adds three small things, all growing along the existing structure:

1. Self-reconciliation queries, on a schedule. The system’s invariants can already be written down: each order’s amount equals the sum of its order items, each orders payment’s receipts are no less than the total of the orders it covers, sold quantity equals the sum of reserving carts and orders. Run them daily on a schedule, silent when green and alerting when red — the cost is a few lines of SQL, and what it buys is turning “should be fine” into “verified fine”.
2. Fix the books with a compensating entry, not by adjusting the DB. Back then undercharging was fixed by adjusting the DB to match — understandable, but strictly speaking it’s rewriting history: six months later nobody remembers why that number looks the way it does. Accounting’s own rule is the right one: don’t erase an error, add a reversing entry. The history of the books is a fact too, and this series’ iron law of append facts, derive status shouldn’t get an exemption at correction time.
3. Manage the export format as a contract. Accounting’s requirements change, and the export is the only interface between the system and them — introduce version numbers, keep samples, put format changes through review, and treat it like an API.

Just those three. No reconciliation platform, no daily full comparison — in a system with exactly one redundancy, building a bigger reconciliation system is fighting an enemy that doesn’t exist.

Reflections

The best reconciliation is leaving the books no chance to be wrong. The need for reconciliation is proportional to materialised redundancy — each extra copy of a derived number is another ledger that can drift and another checking job to write. Leaving derived data on the read side is the cheapest consistency there is: it makes the word “reconciliation” nearly disappear from the system’s vocabulary. Back then it wasn’t that we reconciled well; it was that we designed away the reasons to reconcile, one at a time.

“Nothing went wrong” isn’t “provably nothing is wrong”. This is #15’s bought-on-credit argument replayed on data: our books were clean, but no measurement could prove they were clean — the same silence as the risk chapter‘s “no complaints received”. The design dividend was real, but a dividend needs a report card, and the cost of three invariant queries buys exactly that report card.

The deepest lesson this system taught me is this chapter’s title read backwards. “We never built X, so why did nothing go wrong?” — substitute reconciliation, the state machine, the notification abstraction layer, and the previous fifteen chapters keep showing the same structure: not building it, because the structure made it unnecessary. Engineering maturity isn’t ticking off everything on the list, it’s knowing which items your structure already gave you for free and which you must restore by hand. On the unbundled database laid out across the table, the piece restored most successfully is precisely the one where it looks like nothing was done at all.

Next chapter, a small but beautiful topic: images. Uploading takes an afternoon; deleting takes a lifetime — the life and death of a resource.