Third-Party Payments: Three Textbook Pitfalls, and Terrain That Has None

The last chapter aggregated orders into three layers; now we have to collect the money. Payments are the first time this system hands “being correct” to somebody else — the payment happens at the bank, and you only learn the result across an unreliable network. The textbook warns you of three pitfalls: notifications will duplicate, arrive out of order, and never arrive at all. What this chapter is about is that we fell into none of them — not through luck, but because the design picked terrain that has no pits.

The payment landscape we had

Four payment methods: two banks offering smart transfer and credit card, plus a custom cash payment (operations collecting offline and marking it received). And one hard requirement that existed from the start: partial payment — a transfer caps at 30,000 per transaction, so a checkout of 150,000 means the customer pays across five transfers.

That requirement is a death sentence for an is_paid boolean: money arrives in instalments, across channels, asynchronously. The data model was this:

Three design decisions, and notice how they interlock:

• One fact table per third party. Bank A’s transfers, Bank A’s cards, Bank B, cash — each raw fact stored on its own, rather than crammed into one “generic payment table” with columns forced to fit. Adding a payment method means adding a fact table and an aggregation rule, and “cash” is simply a provider whose receipts are marked by operations.
• The payment’s status is derived, computed at read time. Paid or not = the sum of received facts against the amount due. There is no status column being UPDATEd.
• Do both webhook and polling, writing the same fact tables. The textbook asks you to choose between “live but may miss” and “reliable but slow” — here we take both.

The textbook’s three pitfalls

Let’s be clear about the pitfalls, because they’re real and the industry falls into them daily. In a world where the callback updates status:

1. Duplicate notification: the bank re-sends “payment succeeded” and your status transitions twice — dirty logs at best, duplicate shipments and duplicate invoices at worst. Hence idempotency keys.
2. Out of order: “payment succeeded” arrives before the internal processing of “order created”, or two partial-payment notifications swap places — the state machine goes the wrong way and may never come back. Hence sequence numbers, buffers, compensating logic.
3. Never arrives: the network dropped it, the bank forgot, and your order sits in “awaiting payment” until the end of time. Hence an active query as a backstop.

Three pitfalls, three sets of defences, each one extra code and extra test surface — that’s daily life in most payment integrations.

Terrain that has no pits

In the system we had, the felt experience of those three was “we don’t seem to have run into any problems”. Substitute the architecture above and the reason is plain:

• Duplicate notification? The callback only writes down the fact that “Bank A says this transfer landed” — write the same fact any number of times and the aggregate gives the same answer.
• Out of order? Which of five partial payments arrives first doesn’t matter at all — each fact lands on its own, and the SUM happens at read time.
• Never arrives? The polling schedule fills in the missing fact — and because it’s idempotent, polling overlapping with the webhook is harmless, so the two channels back each other up.

The cost of three defences drops to zero, because there is no status that can be broken. This is the final payoff of the series’ recurring theme of appending facts and deriving state: the same principle taught you at the comment layer that a fact only counts if it’s wired to a path (raw lying in a warehouse, replay never fired), gave you reconciliation at the stock layer, and at the payment layer simply makes the nastiest class of bug extinct.

Reconciliation follows the same structure in tiers: one bank’s payment records carry the orders payment id and reconcile automatically; cash relies on operations marking it — another “automatic takes the bulk, humans take the residue” funnel, isomorphic to the binding funnel in the identity chapter.

An aside: the status column everyone kept wanting to add

This “derive status on read” design wasn’t unanimously applauded — it was held off. More than once the CTO and several colleagues wanted to add an order status column: a status you could query and filter directly, very convenient on a list page. I stubbornly refused, with the same reason every time: that status is by nature an aggregate of other columns and facts — building the column means building a cache. Once a cache exists it has to answer everything a cache has to answer: who updates it, which write paths must remember to sync it, and who wins when it drifts. And we had no performance problem at all. Don’t build a cache until you actually have a performance problem.

That position isn’t a dogmatic refusal to materialise — the stock chapter‘s sold count is a materialised derived value, because it sits on the hot path of ordering (every comment has to check the invariant) and came with hourly recomputation as self-repair. Both decisions used the same ruler: the only legitimate reason to materialise a derived value is a measured read cost; and once you have, you must admit it’s a cache and give it recomputation and reconciliation. “It’s convenient to query” isn’t on the list of legitimate reasons — the price of that convenience is one more column that can lie to you.

Refunds: the system retreats to the observer’s seat

The refund story is a sibling of the state machine lesson. We built a proper refund API — wired to the bank, the flow packaged up — and operations didn’t like using it. They were used to opening the bank’s own console and hitting refund: fast, familiar, with the balance right there.

After some struggle, the decision was to let go. Operations pressed the bank’s button, then marked the order “refunding” — and a scheduled job took over, syncing the refund’s actual progress. The system stepped back from being the executor to being the observer: a human acts, the system reconciles.

That abdication is of a piece with the whole chapter’s architecture: refund progress is just another set of facts, who triggered it doesn’t matter, and the system’s job is to chase the fact down and derive the status correctly. Insisting operations go through your API is, at bottom, building your self-worth on whether other people use your interface; giving the system the ability to catch up with reality is spending engineering where it counts.

Reflections

The best defence is choosing the terrain

Most writing on payment integration teaches you how to guard the three pitfalls: how to design idempotency keys, how to buffer out-of-order events, how often to run compensating queries. All correct — but they’re techniques for fighting on the wrong terrain. The system we had none of those defences and none of those wounds, because “facts are insert-only, status is derived on read” means the pits don’t exist. The long-run lesson for me: when you meet a recurring class of bug, don’t rush to add defences — ask whether there’s a data model in which that class of bug has no foothold. Defences are interest; terrain is the principal.

In a system that handles money, humility beats cleverness

Two retreats in this chapter: no transition rules on status (follow the facts), and refunds not going through our own API (let operations use the bank). Both are the system bowing to reality — and in hindsight both were right. Money flows between the bank, operations and the customer, and your system is only ever one participant. Positioning yourself as “a faithful ledger of every monetary fact” rather than “the only entrance to the money process” makes correctness easier to hold. A ledger doesn’t need to control the world, only to record it faithfully.

”We didn’t run into problems” is the highest grade of result

Payments should be the most incident-prone part of the whole system — the most external dependencies, an unreliable network, the highest cost of getting money wrong — and it’s the calmest chapter in the series so far. Two boring chapters in a row (the last one‘s checkout too), boring for the same reason: get the relationship between facts and derivations right early, and complexity never grows later. Which is also the most unfair thing in engineering: the reward for good architecture is paid out in the form of “no stories” — everyone can see the person fighting the fire, and nobody gets a medal for the fire that never started. A large part of my later job as an EM was learning to see, and reward, the people with no stories.