The Big Picture: Every System One Comment-Placed Order Touches

This is a new series, and the first war story on this blog. At a previous company I built a live-stream personal shopping platform — viewers watch a live stream, type an order into the comments, and behind that sits a whole chain of stock, payments and logistics. This series isn’t a memoir: I want to take everything I know now (after re-reading DDIA, Redis, Kafka and SRE end to end) and fight the same war again — each chapter starts from a real requirement, covers how we actually did it and what we walked into, then gives the design I’d build today. This first post lays out the big picture.

The business: turning a chat room into a cash register

The rules are simple enough:

• The host goes live selling personal-shopping items, and calls out a key for each one (say “2601”).
• Anyone who wants it types 2601+2 in the comments: key plus quantity, and that’s an order for 2.
• If the same person comments on the same key more than once, the last one wins — type 2601+2 then 2601+1 and you’re buying 1.
• An order placed by comment goes straight into the cart, and it reserves stock; every item has a stock cap, and you cannot oversell.
• There are two kinds of cart: the live-stream one reserves stock, the one you fill yourself on the storefront doesn’t. Quantities in both can be changed at any time.
• Comments don’t come from one place: Facebook, Instagram and our own live studio all feed in, and the rules have to apply identically to all of them.
• The nastiest rule: the person commenting may not have an account at all — and stock still has to be held for them.

Each rule on its own is a weekend of work. What’s hard is putting them in the same sentence: three thousand people commenting at once, fighting over 20 units of stock, half of them without accounts, across three platforms — that’s what this system actually looks like.

The journey of an order

Start with the full path from comment to delivery. That path is the table of contents for this whole series:

Every stop has its own trade-offs when you open it up, and the details belong to their own chapters. Here I only want to name what the real problem is at each one:

• Comments arrive: the three platforms are fetched in completely different ways (webhook, polling, our own direct push), so each converges into a unified comment event first and downstream never has to care where it came from. “A repeat comment means the last one wins” looks trivial, but it’s LWW — you have to answer “last by whose ordering?” first.
• Identity: the person commenting is fb:12345, not one of our members. Stock has to be held against that identity, with the account showing up later and claiming the order back. This is the most underestimated chapter in the series.
• Stock reservation: never overselling is the one iron rule of this system, and underneath it’s a fight to defend an invariant under concurrency. Whatever gets reserved has to be releasable — expiry, cancellation, quantity changes; miss one and stock bleeds slowly.
• Checkout and payments: the two carts converge here, and the payment webhook will arrive twice, out of order, or never at all. Every lesson from unreliable networks gets examined here.
• Pre-shipment: the personal-shopping speciality — orders from several live streams get consolidated into one parcel, waiting for everything to arrive, one person’s many orders boxed together. The stock ledger only really settles at the moment of picking.

The three genuinely hard things

The feature list isn’t hard. What’s hard is three properties that cut across the whole chain:

1. The peak isn’t a curve, it’s a wall. In the three seconds after the host calls a key, thousands of comments land at once — this isn’t a normal e-commerce “campaign traffic ramp”, it’s a thundering herd triggered by one sentence (the flash-crowd chapter is all about this).
2. No overselling — and specifically, not under concurrency. Twenty units, three thousand people; any naive “check then decrement” will oversell. That invariant has to be guaranteed at the architecture level, not by being careful.
3. Three ledgers have to line up. Stock, orders and payments live in three different systems, and given time they will drift. Once they have, you need to be able to answer “which one is right?” — a system with no source of truth can only reconcile by guessing (the reconciliation chapter is all about this).

The architecture I’d build now: write down the fact first, derive everything else

The system we had was already at this doorstep: comments got pulled in, cleaned, landed in the DB, and consumed in batches by a job. Starting over, I’d change one thing — take the thing we treated as a buffer and promote it to the centre of the architecture: a comment isn’t input waiting to be processed, it’s a fact that must be kept forever:

Three claims in that architecture, and they’re the recurring themes of the whole series:

1. Write down the fact first, talk business second. A comment is appended to the event log the moment it arrives, and the business logic is a consumer of that log. That one step solves three things at once: the peak (a write is just an append, with the queue absorbing the spike — Kafka exists for exactly this), debugging (every input leaves a trace and can be replayed), and reconciliation (you now have an answer to “which one is right?”). This is the field version of DDIA Part III: a fast-write source of truth plus fast-read derived views.
2. Stock is a reservation model, and the holder is an identity, not an account. “Holding stock” really is a reservation: held for fb:12345 as an identity, claimed once an account is registered. Identity coming before account is where this business differs most from ordinary e-commerce.
3. Assume every external integration will repeat, reorder and disappear. Platform webhooks, payment callbacks, courier status — idempotency isn’t a nice-to-have, it’s day-one foundation.

The series map

The series follows the journey of an order, with operations and evolution added as depth (chapters are still growing; the live series list is authoritative):

• Opening and foundations: the big picture (this post), the stack and the CI/CD starting position.
• The transaction spine: comment-to-order, identity and accounts, stock, cart and orders.
• Money and goods: third-party payments, pre-shipment handling.
• Operations: the host console, permissions, promotions and amounts, notifications, risk and blocklists.
• Cross-cutting and ops: the flash crowd at open, running production without an SRE, reconciling the three ledgers.
• Evolution and endgame: breaking the monolith into microservices, what if it became SaaS, the EM’s view of a small team, and finally “Re: if I really started over”.

Reflections

The biggest mistake back then wasn’t the technology choice

It was treating “landing the data” as a buffer instead of a source of truth. We got it half right: comments were pulled in, cleaned, written to the DB, then consumed in batches — already half an event log. But the raw text was thrown away after cleaning, so any comment the cleaning rules mishandled was gone for good; and a comment whose batch failed was simply skipped, with no trace and no recovery path, so that customer disappeared in silence. It was a deliberate trade at the time: we’d rather drop an order than slow down the stock figure the host was watching. And we didn’t even keep the speed — at peak, consumption fell behind, comment-to-confirmed-order could stretch to minutes, the host’s mental model of stock stopped matching the system’s, and the complaints grew straight out of those minutes. I’d still choose fast if I did it again — but fast can be bought with “process it later”, never with “throw the fact away”: keep the fact and a late order can still be recovered, a complaint can still be traced to a person; lose the fact and you don’t even know who to apologise to.

Invariants matter more than features

Get a feature wrong and you fix it and ship again; oversell and you’ve sold goods that don’t exist, which means apologising and refunding one customer at a time, with trust spent that doesn’t come back. So the three rules in this system — never oversell, money and ledgers agree, integrations are idempotent — are what I’d put on page one of the design document on day one if I started over. Not because it’s elegant, but because the cost of breaching those three is paid in reputation. That’s the core of what I want this series to say: what’s hard about an e-commerce system isn’t the features, it’s the invariants.

Why write it as a “do-over”

Because “how would you design it if you started over” is my own favourite question to ask in interviews, and I’ve found the most honest way to answer it is to take a system you really built and really got wrong. Every chapter from here has two voices: how we actually did it and why, then how I’d do it now and why. The interesting part was never the model answer — it’s the gap in between. That gap is what I actually learned these past few years.