Risk Control and Blocklists: Not an Eviction Notice, a Credit System

Let’s state the threat model first. In this system, calling an order in the comments holds stock immediately while payment happens at the end of the round — with days of trust in between. So the malicious play is simple: call it, hold it, don’t pay. Stock sits squatted for up to a full week (a round’s length), customers who actually want it can’t get it, and the host’s goods are stuck with phantom orders — malicious ordering is a DoS on stock, and it costs nothing to launch: a few keystrokes.

Stock is this system’s common resource, and risk control is its governance. This chapter is what that governance looked like — and it’s far more refined than the word “blocklist” suggests.

Graduated sanctions: from one confirmation button to a permanent ban

Getting caught not paying isn’t an instant execution, it’s four escalating tiers:

The first tier is the soul of the whole ladder. It isn’t a punishment, it’s a signature: next time you open the site a warning dialog appears, you press confirm and you’re unlocked on the spot — nothing lost, but “the rule has been communicated” is now done. I used to think false positives (the good customer who genuinely forgot, or went abroad unexpectedly) had to be rescued by an appeals process; this design gives a higher-order answer: make the first tier’s penalty so cheap that a false positive doesn’t matter — an innocent person’s entire cost is pressing a button, without even contacting support. Detection doesn’t have to be accurate, because the harm of judging wrong has already been absorbed structurally.

And repeat offenders need no separate detection — they climb the ladder by themselves: a month, three months, a permanent ban. That progression of warning → light penalty → heavy penalty → expulsion has a name in governance literature, graduated sanctions, and it’s a classic design for managing a common resource. Nobody here had read any of that literature; it grew out of business instinct.

Ban the call-out, never the card

What do you actually lose when banned? This is my favourite cut in the chapter: a banned user only loses the ability to call orders in a live stream — paying by card on the storefront is as welcome as ever.

Unpack why that’s right. This system has two kinds of transaction:

• Calling an order is a credit transaction: a comment holds stock now and payment comes at the end of the round — for those days in between, the platform is extending you credit.
• Paying by card on the storefront is a cash transaction: pay first, then goods — requiring no trust at all.

What someone who maliciously doesn’t pay abuses is the credit in the former — so what gets taken away is precisely that privilege. Someone whose credit is blown isn’t barred from doing business, they’re limited to cash — several centuries of banking logic, reinvented inside live commerce. The commercial dividend is real too: you still earn a repeat offender’s money, they just pay up front; and the road back for the genuinely reformed is always open, by building a record on the storefront. The granularity of the punishment matches the granularity of the trust abused — punish exactly what was done, no more and no less.

The machinery: one table, one cache, a row of buttons

Every part of the enforcement layer has a relative you’ve met in an earlier chapter:

• The banned user table uses content type + object id — a generic foreign key making its third appearance in this system (cart source, and now the subject of a ban): block an fb user or block an account, one table covers both. Guilt by association is at account level: ban the account and every identity bound to it is covered; and binding updates Redis on the spot — bind a new identity to a banned account and it inherits the record immediately. Of course, identity’s inherent limit remains: a brand-new face (a fresh Facebook account) sometimes simply can’t be stopped — the plain truth from chapter 4, since risk control can only raise the cost of anonymity, never abolish it.
• Redis does only the fast check, the fact lives in the DB — the correct posture for a cache: every comment has to ask “is this person listed?”, a high-frequency small lookup that a batch can’t amortise, so it lives in Redis; on restart it’s rebuilt from the DB, and after a round settles it’s written with a pipeline in batch (a whole season’s list update in one round trip), leaving only a brief window at settlement.
• One-click blocking from the comment waterfall — on the host’s and assistant’s dashboard, the enforcement button sits right next to the blocklist tag: hecklers and abusers dealt with live, without waiting for a settlement. The intelligence card and the enforcement desk are the same screen.
• There are two ways out: expiry (the one-month and three-month locks), and support lifting it by hand in the user management interface (which can filter by banned status directly — internal-tool UX for the Nth time).

The rebuild: give risk control eyes

The rebuild list here has one item, and it’s a type you haven’t seen in any earlier chapter: measurement.

After this went live, “we didn’t get any negative feedback afterwards” — but we never measured anything about it. How many malicious orders were blocked? How many paid up after a first-tier warning (the reform rate)? Were good customers wrongly hit with three-month locks (the false-positive rate)? How many permanently banned people came back with a new face? None of it is known. “No complaints” is silence bias, not evidence — the most likely reaction from a wrongly banned customer isn’t an appeal, it’s quietly never coming back.

In hindsight, risk control is the one subsystem here that was built without knowing whether it worked: stock had invariants guarding it, payments had reconciliation, notifications had a delivery column — risk control had nothing. The rebuild list is one line: how often each tier fires, the payment rate after a first-tier warning, the repurchase rate after a lock lifts, the return rate of repeat offenders — install the eyes first, then talk about tuning. That was a luxury back then and is common sense in the data-engineering world I moved into later: a mechanism with no measurement is one where even “is it still working?” is an article of faith.

Reflections

Designing the punishment matters more than designing the detection

The industry talks about risk control with nine tenths of its effort on detection: better models, more signals, faster interception. This system put its weight on the other end — the structure of the punishment: a first tier so cheap that a false positive doesn’t matter, an escalation that makes repeat offenders surface by themselves, and punishing exactly what was done so the innocent parts aren’t touched. The result is that detection can be dumb (just check whether they paid — not even a model) and the system still works. That ordering is worth pondering for anyone doing risk control: detection accuracy is an incremental arms race, and punishment structure is a one-off design — when the punishment structure is right, detection only has to be roughly right; when it’s wrong (one strike and a permanent ban), even perfect detection is manufacturing injustice.

Governance wisdom grows out of business instinct

Graduated sanctions, signing for the rules, punishing exactly what was abused, association and expiry — every design in this chapter has a counterpart in the literature on governing common resources, and nobody involved had read any of it. They were instincts ground out by hosts, operators and support across one stream after another, and engineering only translated the instincts into a table and a cache. This is the third appearance of the same thing as greedy beating the optimum and the state machine getting ripped out: the floor’s understanding of how people behave routinely runs ahead of an engineer’s understanding of how the system should be designed — a good system designer doesn’t invent rules, they hear the rules the floor is already using and make them executable and traceable.

The end point of risk control is keeping the person

“Ban the call-out, never the card” hides a value that’s easy to miss: the goal of risk control isn’t expulsion, it’s repairing trust. Every tier of the ladder keeps a way back — press a button, wait a month, rebuild a record on the storefront; even a permanent ban leaves the support door open. Against the many platforms whose approach is detect an anomaly → ban permanently → no appeal, this design takes “people make mistakes, and people come back” as the default. And that’s where the loss stings most: the reform rate — the only evidence that this value was ever redeemed — is precisely the number that went unmeasured. The first thing those rebuilt eyes should look at is that one.