The Moment of Opening: The Three Seconds After the Host Calls a Key

The transaction and operations stories are done; the next few chapters are about things that cut across the whole system: peaks, operations, reconciliation. Starting from the origin point of all live commerce — the three seconds after the host calls a key. This chapter has real numbers, the most painful incident of all, and a war story running from being flattened all the way to a “cloud bodhisattva”.

The shape of a peak: a wall, then a run of waves

Numbers first. A big stream had up to 13,000 viewers online, with an ordinary peak around 9,000. While the host is introducing a product, comments average 10–20 a second; at the instant they call the key and open bidding — 200 a second.

Two things about that shape: it’s a step, not a ramp — one sentence from the host and traffic jumps 10–20× within a second, faster than any autoscaling can react, so capacity has to be provisioned for the peak and shaving it can only come from structure; and it’s a pulse train, not a single peak — a close averages three minutes, and during a long one the host drops promotions at random moments and traffic surges again. Adaptive fetching (speed up with the paging key on a surge, slow down when idle) looked like instinct at the time; against this shape it’s precise: fixed-rate polling loses at both ends of a pulsing load.

One more thing worth saying up front, because it’s the chapter’s foundation: 13,000 viewers online, but the write peak is only 200 a second — at peak everyone is in the comments, and the site’s writes are actually modest. If this were a traditional e-commerce flash sale, 13,000 people would be 13,000 concurrent checkout requests landing straight on your API; “buying by comment” compresses thirteen thousand purchase intentions into one linear text channel, with Facebook’s chat infrastructure absorbing the fan-in for free, leaving your system a single 200/s stream to digest. The business model shaves its own peak — an interaction that looks primitive turns out to be brilliant traffic design, and nobody thought about it that way at the time.

Two fates: batch silts up, sync blows on contact

With the shape of the traffic established, here’s the chapter’s thesis: under a peak, this system’s components have two fates.

That batch column is the third time batching saved this system: comment orders on the FSM batch (minutes of backlog at open, but no crash), win notices on the FB batch API, and even the host dashboard’s comment waterfall pushed in batches — 200 a second, absorbed. An overloaded batch system’s symptom is silt: the queue lengthens and latency grows, but it doesn’t fall over; it pays in latency rather than availability.

That also explains something interesting in hindsight: this system had no degradation features at all — not an oversight; the line at the time was “we haven’t had any incidents yet, what would we degrade?”. Half right; the other half is deeper: a degradation switch is a necessity for synchronous systems, because their overload is an avalanche; a batch system’s degradation is built in, silting rather than falling by nature. Everything that actually blew up was in the sync column — which brings us to the war stories.

The most painful one: a poison pill

First, the batch side’s only serious wound, which wasn’t inflicted by traffic but poisoned by a single row of data.

The system wasn’t fully launched yet — some acceptance testing was still outstanding, but the host’s contract with the original third-party software company collapsed early and the system was pushed straight into combat: the launch date was never ours to set. Then during one stream something strange happened: plenty of comments, and orders barely trickling in. The host was furious.

The culprit was one line at a boundary: within a batch of 200, one comment failing to parse unexpectedly aborted the whole batch — one weird comment dragging 199 innocent orders down with it. The stream-processing world calls this a poison pill, and our version hurt especially because of the blast radius: the unit of failure was the batch, not the row. The symptoms were deeply deceptive too: ingestion was entirely healthy, the comment waterfall kept flowing, and only the conversion rate quietly went to zero — the hardest kind of failure to catch on a monitor is exactly this, where nothing is broken and there’s simply no output.

The fix was that “second-best” measure from the comment chapter: skip the failing row and keep the rest running — my memory is a little distant, but it should have been changed right after this incident. Put in evolutionary order, the “skipping” we criticised was in fact the bleeding stopped from something far worse: the whole batch dies (the incident) → skip the single row (the fastest tourniquet) → dead-letter recovery (step three of the rebuild). Shrink the blast radius first, restore completeness later — the order is right, only step three never got finished.

Read-path war stories: from a fat API to the cloud bodhisattva

The sync side’s story is a complete causal chain, worth telling in order.

Link one: an API containing everything. A user logs into the home page and calls the “round list” API — and the response is detailed information, right down to product data. What flattened a single process was never the number 13,000, it was the weight of each request: deep queries, heavy serialisation, a fat payload, with every logged-in user pulling the whole catalogue.

Link two: emergency tourniquet. The single process came down immediately, traefik went in front, and four API services came up — it held. But “there was too much to do, so once it held we left it” — the fat endpoint stayed exactly where it was. That’s the standard life cycle of technical debt: what separates the tourniquet from the cure isn’t ability, it’s a priority that never makes the list.

Link three: the next load test was run for free by an attacker. One day a DDoS blew it open — an hour of being unable to move at all. We rushed GCP’s Cloud Armor into place, and it did nothing; the bill, on the other hand, went up $1,000 in an hour, plus what Cloud Armor cost on top. GCP really does know how to earn.

Link four: the cloud bodhisattva. Finally we moved the domain name layer to Cloudflare — and it was fine, and free. Truly, a cloud bodhisattva saving the mortal world.

One lesson per link: an API’s schema design is the front line of capacity engineering (the cheapest scaling is not sending fields nobody wants); the consequence of “it holds, leave it” is letting an attacker prioritise your technical debt for you; and the one nobody spells out — your defence must not be billed alongside the attack. Per-request-priced L7 defence under a DDoS amplifies your bill on the attacker’s behalf; Cloudflare absorbs the traffic at the DNS/edge layer and the free tier is enough — the right place for defence is the edge, not the door of the origin.

What a rebuild would do

1. Hold the line on list APIs. A list returns only the fields needed, under a convention that’s consistent site-wide; stay RESTful and don’t cram every odd thing into one response. This one has to be won in requirements meetings, not in the server room — one extra field is cheap for the front end, and 13,000 people each taking a copy is a capacity incident.
2. Put the domain on Cloudflare on day one. Leave the specialised things to specialists rather than letting an all-in-one vendor take a cut for something that may not even work well — DDoS protection, CDN, a free tier, all at once.
3. Finish step three of the poison-pill answer. After skipping the single row, failed events go to a dead-letter queue for replay (the comment chapter‘s rebuild lays it out) — blast radius shrunk to one row, completeness caught by the recovery path.
4. Put a gauge on the batch backlog. “Silts but doesn’t fall” only works if you can see the silt — batch lag is the single health metric that matters in this architecture, and it should be on a screen before the host goes live, not discovered when somebody in the comments asks why their notice hasn’t arrived.

Reflections

Overload always gets paid for; the architect chooses the currency

There’s no free lunch under a peak: once load exceeds capacity, the system pays along some dimension — a batch system pays in latency, a synchronous one pays in availability. The entire difference between this chapter’s two storylines is the currency: comments backlogged for minutes cost a few complaints and the system lived; the fat API blew on contact and took the whole site with it. So “batch everything batchable” isn’t a performance optimisation, it’s choosing how you go bankrupt — a latency debt can be paid in instalments (work the backlog down), an availability debt is due in full immediately. When designing a system, ask yourself: for each path, which currency do I pay in at overload? Whichever one you can’t afford, convert it into one you can.

A peak is the shape of the business model, not of the traffic

The 200/s wall, the pulse train of promotions, two channels overlapping — those aren’t “traffic characteristics”, they’re the rhythm of the business projected directly onto the system: the host calling a key causes the wall, dropping promotions causes the waves, ordering by comment causes the peak-shaving. Understanding how the business runs before doing capacity planning beats any load test — and conversely, the first lesson of a cascading failure is the same: a system’s failure modes are also decided by the shape of the business. That “the business model shaves its own peak” dividend was one we enjoyed for a long time before noticing it existed — sometimes the best architectural decision is one the business made for you.

The bill is an attack surface too

During that DDoS hour, the system being frozen was one kind of wound; $1,000 extra on the bill was another — and the second kind keeps going after the attack ends, because the service you bought to defend yourself is also billed by traffic. Security design in the cloud era has to include the cost model: an attacker’s cost approaches zero, and if your defence is priced per request you’ve already lost economically. Keeping an attack off your billing meter (edge layer, flat rate, free tier) matters as much as keeping it out of your system — half of security architecture is financial architecture.