Tech

the peak seven people · shipping daily that day CTO: development is paused contract redone · redundancies a month later three people left me · one front end · the CTO new direction purchasing system for external orders the split bidding out · monolith demoted Background noise: the storefront is still up, but there are no customers

Microservices for Three People: After Development Was Paused

· tech · 26 min read · 📚 Re:Building a Live-Commerce Platform from Zero #19

The previous chapter ended by saying this team's end was nearer than anyone thought. This chapter st…

#war-story#live-commerce#microservices

storefront line me (backend lead) backend ×1 front end ×2 CTO: full-time PM no code in the storefront period engineering output = 4 engineers bidding line (separate) backend ×1 · front end ×1 dedicated PM ×1 honestly: I don't know this line well (early) 1–2 contractors — a tourniquet for the single-backend period fully remote · the split line = the org line (foreshadowing the next chapter)

Six Engineers Running at the Speed of Twenty

· tech · 23 min read · 📚 Re:Building a Live-Commerce Platform from Zero #18

The system's story closed with the previous chapter; the remaining chapters are about the people who…

#war-story#live-commerce#engineering-management

the live floor assistant screenshots stream pastes into a dialog, confirms front end: optimisation crop square · convert to webp for experience and bandwidth back end: guarantee always re-encode + thumbnail original bytes never land GCS image + thumbnail two files transcoding-as-validation: a file that won't decode fails by itself image_metadata: path · content type · object id The DB stores the path (fact); the API resolves a URL (derivation) — images are served from GCS behind a CDN What the front end does is optimisation; what the back end does is guarantee

Easy to Upload, Hard to Delete: The Life Cycle of Images and Resources

· tech · 27 min read · 📚 Re:Building a Live-Commerce Platform from Zero #17

A detour into the least glamorous topic here: product images. "Isn't that just file upload?" — this …

#war-story#live-commerce#system-design

Inside a database Our system WAL: write the log before anything else comments cleaned, then appended and landed log consumer: digest the log, build indexes FSM batch digests comments, builds carts materialised view: precomputed and stored sold-quantity count (the only materialisation) view: computed at read time payment / order aggregate status: derived on read redo log: a replayable history of change allocation log: rebuildable end to end built-in scheduler: vacuum, checkpoint heartbeat scans: chasing payment, settlement repair: fix it back from the facts hourly recomputation of sold quantities The price of unbundling: restoring, one by one, the transactional guarantees a database gives you for free And this chapter answers how we restored the one called "consistency"

Reconciliation: We Never Built It, So Why Were the Books Still Right?

· tech · 22 min read · 📚 Re:Building a Live-Commerce Platform from Zero #16

In the plan this chapter was called "the three ledgers: stock, orders, payments", and it was going t…

#war-story#live-commerce#data-consistency

13,000 viewers online one VM traefik API+WS API+WS API+WS API+WS Celery: heartbeat the scheduling heart Celery: fetch comments only 1 worker Celery: async tasks 10 workers · acks_late Redis RabbitMQ Deliberately doing it on as little as possible — a philosophy, not a budget Cloud SQL 8 cores · no self-run DB

The Years Without an SRE: A Backend Lead's Launch Diary

· tech · 26 min read · 📚 Re:Building a Live-Commerce Platform from Zero #15

The previous chapter was about how to fight the peak; this one is about the person doing the fightin…

#war-story#live-commerce#sre

200/s 10–20/s introducing the product bidding opens: 10–20× in one second promos dropped during the close → a run of waves one close: 10 seconds at the short end, 3 minutes on average 13,000 viewers online; a second channel opening bidding at the same time stacks to ~250 comments/s at worst

The Instant the Sale Opens: The Three Seconds After the Host Calls a Key

· tech · 26 min read · 📚 Re:Building a Live-Commerce Platform from Zero #14

The transaction and operations stories are done; the next few chapters are about things that cut acr…

#war-story#live-commerce#scalability

1st offence a warning dialog on arrival press "confirm" to unlock sign for it · false positive ≈ 0 2nd offence locked for one month expires automatically 3rd offence locked for three months expires automatically 4th offence permanent ban only support can lift it Reoffend and you climb a tier — grading makes false positives cheap and lets repeat offenders climb into the heavy penalties

Risk Control and Blocklists: Not an Eviction Notice, a Credit System

· tech · 21 min read · 📚 Re:Building a Live-Commerce Platform from Zero #13

Let's state the threat model first. In this system, calling an order in the comments holds stock imm…

#war-story#live-commerce#risk-control

message severity → private reply automatic · free · high volume win notice + binding token policy wall: one comment gets only one reply email routine · internal notices async completions go here Google Workspace, off the shelf downside: it sinks phone (support) manual · priciest · always lands final notice: before the money is cleared and they're blocklisted source: phone numbers for free shipping SMS: OTP + one final chase per round whoever the SMS doesn't move gets a call Channel cost tracks message severity; and reach isn't granted by an API — the product bought it

The Notification System: Two Columns, One Scheduled Job, and a Phone Call

· tech · 22 min read · 📚 Re:Building a Live-Commerce Platform from Zero #12

What "a notification system" looks like in a textbook: a notification service, a message queue, a te…

#war-story#live-commerce

With a coupon: one coupon = a combination of three parameters Effect amount off over a threshold free shipping over a threshold Threshold how much spend triggers it + product allowlist (what counts) Scope one round / several rounds all rounds combined e.g. "300 off over 3000 · round A only · selected products" No coupon: multi-buy offers buy N, get M free one product can carry several the ordering story is the next section Experiment layer: Django admin the unusual buy-A-get-B combos applied urgently, experimentally promoted to a real rule once it holds

Promotions and Amounts: A Miscalculated Discount Is Harder to Find Than Overselling

· tech · 38 min read · 📚 Re:Building a Live-Commerce Platform from Zero #11

Overselling explodes and payments scream; a miscalculated discount makes no sound at all — a custome…

#war-story#live-commerce

Act I Django group + permission model-level CRUD machine granularity, no semantics Act II (+1 week) permissions stuffed into the JWT wrong granularity, in concrete a change = a re-login Act III move to role-based semantics finally right but an orthogonal axis appears and roles start exploding Landing stackable capability roles cost monitor multiply → add Every step was locally reasonable: built-in (saves time) → into the JWT (stateless) → roles (semantics) The error is in none of the steps — it's never having asked what this system's natural unit of permission is

Permissions: Who Can Press Which Button — We Reworked It Three Times

· tech · 31 min read · 📚 Re:Building a Live-Commerce Platform from Zero #10

Permissions are, by my own admission, one of the parts we didn't get right back then: reworked three…

#war-story#live-commerce#authorization

Host view only, no actions comment stream (with blocklist tags) units sold who ordered viewer count (FB API) data sets the rhythm Live assistant the live console enter product info (ahead or on the fly) open · close bidding re-open add stock acts for the host Operations its own pages close a round shipping fees goods-in allocation owns round rhythm Support handles exceptions adjust carts clear carts multiple binding catches the residue Engineers Django admin safe console Celery tasks odd requests (rough-draft lab) uncertain starts here One set of facts and APIs underneath — the same data through five different windows

The Host and Operations Back Office: There Is No Such Thing as "the Back Office"

· tech · 18 min read · 📚 Re:Building a Live-Commerce Platform from Zero #9

The first eight chapters were the skeleton of a transaction: comments in, stock held, money collecte…

#war-story#live-commerce

sales stock (the promise) cap · sold count — guards overselling at order time lives in millisecond transactions physical stock (reality) goods-in appended — operations logs what arrived lives on a day-scale logistics rhythm Books drift by nature: 100 agreed, 80 delivered allocation system physical stock → assigned to orders; operations decides how allocation log appended — rebuildable from events order fully allocated → shipping note (items go together, never split) 7-Eleven pickup API · customer picks a store online home delivery (own courier) CSV exported by hand to the courier

Pre-Shipment: You Sell a Promise, You Ship Reality

· tech · 24 min read · 📚 Re:Building a Live-Commerce Platform from Zero #8

The money is in; time to ship. This chapter is where the system meets the physical world — comments,…

#war-story#live-commerce

orders payment status not stored, aggregated on read: paid = SUM(received facts) ≥ due Bank A · smart transfer fact table Bank A · card fact table Bank B · payment fact table cash receipt marked by operations 150,000 cap 30k each = 5 facts webhook (live) the bank pushes it polling (backstop) scheduled queries Both channels write the same fact tables — facts are idempotent, so overlap is harmless and each backs the other

Third-Party Payments: Three Textbook Pitfalls, and Terrain That Has None

· tech · 21 min read · 📚 Re:Building a Live-Commerce Platform from Zero #7

The previous chapter aggregated orders into three layers; now we have to collect the money. Payments…

#war-story#live-commerce

reserving cart (live) can span several rounds / hosts non-reserving cart (storefront) same cart item table, polymorphic source merged checkout orders payment — the unit of paying pay once across rounds · aggregates 3rd-party payment facts (next chapter) order (round A) fulfilment unit · round coupons here order (round B) split by round order (round C) each on its own shipping rhythm order item — accounting unit: amount frozen at sale; invoices/refunds follow it

From Cart to Order: The State Machine the Hosts Ripped Out

· tech · 28 min read · 📚 Re:Building a Live-Commerce Platform from Zero #6

The last stop on the transaction spine: how a purchase travels from cart to order. Comments made it …

#war-story#live-commerce#system-design

product price, product info changes all through a stream 1:1 inventory table (hot data, its own table) stock cap — a host restock adjusts only this cart qty (reserved) order qty (paid) payment: cart → order, moved inside one transaction invariant: cart + order ≤ cap (sellable is derived, never stored) FSM batch orders · LWW edits customer adjusts quantity support clears · adjusts carts assistant / operations restock · close round Four writers — but both counts are fully recomputed hourly from cart/order items (a derived value healing itself)

Stock: Never Oversell — This System's One Iron Rule

· tech · 27 min read · 📚 Re:Building a Live-Commerce Platform from Zero #5

With comments parsed and orders hung off an identity, the main line reaches the heart: stock. This s…

#war-story#live-commerce

identity: a fact fb user (PSID) created on the spot, at comment time ig user own-studio user account: an aggregate account appears only after login claims identities' orders, 1:N binding (can be multiple) cart item (holds stock) attached to the fb user, not the account fbmsgtocartitem msg id · bidding key id aggregate view: claim it, don't move it

Identity and Account: Who Exactly Is the Person Commenting?

· tech · 26 min read · 📚 Re:Building a Live-Commerce Platform from Zero #4

The FSM in the previous chapter worked out "who bought what" — except that "who" is only a string of…

#war-story#live-commerce#identity

FB (100) · poll IG (10) · webhook Own (1) · push Fetch: per-source FB polling: busy→fast, idle→slow all into one table Clean → land append in a unified format dedupe: source+message id batch 200 · FSM parse → look up bidding key decrement sold + identity raw is stored · FB re-fetchable — never used failure → skipped, no recovery path All for the freshest stock the host can see; the price: minutes of lag at peak, orders lost in silence

Comment-to-Order: Turning a Chat Room into an Order Channel

· tech · 42 min read · 📚 Re:Building a Live-Commerce Platform from Zero #3

With the big picture and the opening move in place, here's the first battle on the main line: how on…

#war-story#live-commerce#fsm

Synchronous · ms Django API request in, response out cart · checkout · members Real-time · push WebSocket viewer comments → live to the host dashboard Async · sec to min RabbitMQ (the pipe) Celery workers fetch comments · FSM orders · invoices email · order exports PostgreSQL the single truth: orders · stock · members Redis speed: fast banned-user checks One specialist per timescale, with one truth and one speed underneath

The Opening Move: Five Components and One CI/CD Pipeline

· tech · 16 min read · 📚 Re:Building a Live-Commerce Platform from Zero #2

With the big picture laid out, and before we get into the battles over comments and stock, let's ope…

#war-story#system-design#django

Comments arrive FB / IG / own studio Unified comment event one adapter per source Parse key+n same person → last wins Identity order without an account Reserving cart stock −n · no overselling Checkout two carts merge Payment gateway webhook · idempotent Pre-shipment consolidate · to courier unpaid past deadline → release stock Every stop is a chapter: comment intake, identity, stock, cart, payments, pre-shipment

The Big Picture: Which Systems a Comment-to-Order Sale Passes Through

· tech · 22 min read · 📚 Re:Building a Live-Commerce Platform from Zero #1

This is a new series, and the first war story on this blog. At a previous company I built a live-str…

#war-story#live-commerce#system-design

Traditional: one box, everything bundled storage engine indexes cache mat. view log ↓ unbundle: each feature handed to a specialised system ↓ log as hub (Kafka) — the glue that sets order OLTP DBstorage + transactions Elasticsearch= the index Redis= the cache warehouse / Gold layer= materialized view every system is a follower of the log, consuming in the same order → each consistent Your data platform = a database "turned inside out"; what holds it together is the log

The Future of Data Systems: The Unbundled Database, Kappa, and End-to-End Correctness (Finale)

· tech · 13 min read · 📚 Designing Data-Intensive Applications — Reading Notes #12

The final chapter. The first eleven took storage, replication, partitioning, transactions, consensus…

#distributed-systems#book-notes#data-engineering