Notifications: Two Columns, One Scheduled Job, and a Phone Call

What “a notification system” looks like in a textbook: a notification service, a message queue, a template engine, multi-channel SDKs, a retry-with-backoff framework. The version in this chapter is two columns, one scheduled scan, a ten-second rule, and a phone call from support. It’s too small to look like a system — and it delivered 99% of the notifications, with somebody catching the other 1%.

The channel ledger: cost aligned with severity

Take stock of the channels first. Notifications here travelled four roads, and the more severe the message, the more expensive the channel:

The line worth pausing on is chasing payment, which is two-stage. At the end of a round, one chase SMS goes out — SMS has an API, but it’s used exactly once per round, taking the discipline of rationing a scarce channel to its extreme. Whoever the SMS doesn’t move goes to a support phone call: pay now or the order gets cleared and you go on the blocklist. That kind of final notice is too important for any automatic channel — private reply has a policy wall (Facebook allows one reply per comment, and the win notice already used it), email sinks, and the only message that always lands is a phone call. And SMS or phone, both need a number — which is why we ran a campaign giving free shipping for adding a phone number, and plenty of customers did. Buying a high-reach channel with shipping costs, and paving the way for SMS OTP while we were at it — the right to reach someone is a scarce asset; it doesn’t come with an API, a product trades something for it. It’s one of my favourite pieces of growth design in this whole system.

A notification queue with no queue

How does a win notice go out? The textbook draws a message queue. The answer we had: you don’t need a queue, because the fact table is the queue.

The flow fits in a sentence: an order call upserts fbmsgtocartitem, a scheduled job scans that table for rows that are closed, unsent and under 5 retries, calls Facebook’s batch API to send the win notice (with the binding token attached), and writes back either success or an incremented retry count. Success, or five retries, is terminal — bounded at-least-once, with no zombie task retrying forever.

The virtue of that design only shows against the textbook version: most teams would stand up an MQ plus a notification service plus a delivery status table for this requirement — three new components, three new consistency boundaries. Here it’s two columns: the queue is the WHERE clause of a scan, the delivery ledger is the columns themselves, and “did this customer get the notification?” is one SQL query rather than a trace across three systems. fbmsgtocartitem now holds five identities: order provenance, the target of an LWW upsert, a notification queue, a delivery ledger and a reconciliation anchor — five roles in one table without strain, because each role reads and writes only its own columns, and they all share one granularity of fact: one comment, one row.

War story: the rate limit Facebook’s docs got us into

The only incident in this chapter. We started by sending notifications one API call at a time — and then got rate-limited, win notices jammed up, customers asking in the comments why theirs hadn’t arrived, and the host furious.

The galling part: the volume we were sending was well under the rate limit in Facebook’s documentation. We checked against the docs for ages, weren’t over it, and were still being limited. The fix was switching to the batch API to send in one packet, after which it was stable at 99% delivery — with the remaining 1% being accounts that simply could not be reached (Facebook-side reasons), picked up by support.

Two lessons: an external platform’s documentation is a reference value, not an SLA — the real limit can only be measured, especially when your traffic pattern (a burst at the instant bidding closes) isn’t what the platform imagined; and “couldn’t be sent” has to be a visible state — that success column gives the 1% residue somewhere to be queried and someone to catch it, instead of evaporating in silence. Which answers the comment chapter‘s “failures are skipped” lesson from across the series: what we learned the hard way on the comment side, we got right on the notification side.

Three small rules that each save a subsystem

The rest of this chapter is three rules that each avoid a subsystem:

• The 10-second rule. Any operation taking more than 10 seconds becomes an async task that emails the operator when it’s done (exporting a round’s orders, say). “Should this be async?” turns from a case-by-case architecture discussion into one constant rule, and internal users learn the mental model: long things arrive in your inbox. Incidentally, staff log in with company email + OTP anyway — notification and authentication share one chain of trust, and there are no passwords.
• The chase list isn’t a feature, it’s a query. Support needs to phone people about unpaid orders — where does the list come from? The back office’s order and cart management pages support composable ad-hoc search (Django Ninja composing filters), so “unpaid cart items in a round that’s about to close” is a set of query conditions. Make the general query mechanism good enough and the demand for bespoke list pages disappears by itself — mechanism to the system, policy to people: the system supplies the query, and support decides who to call and in what order.
• In-app notifications: not built. In-app notifications imply a whole set: unread state, a notification list, read receipts, push — while email is already sitting there in Google Workspace. “Not worth building yet” is judgment this team showed over and over: build abstractions and infrastructure when the second real requirement shows up.

There’s one “don’t notify” worth collecting too: on a re-call, customers whose old orders were wiped get no separate notification — the host saying it on air is the notification, and every kindness the system adds leaks a little of the stream’s urgency; whoever wins the item again gets the usual win notice. A notification system’s boundary isn’t only “what to send”, it’s also “what to deliberately not send”.

What a rebuild would do

This is the shortest rebuild list in the series: keep it essentially as is. Two columns as a delivery ledger, a queue made of a table scan, tiered channels, the 10-second rule — all still the right size today. If pressed, two small things:

1. Make “couldn’t be sent” a default filter on the support page. The residue is already queryable (the success column is there); a rebuild makes it one click — so the 1% list arrives in front of support instead of waiting for somebody to think of querying it.
2. Abstract when the second channel arrives. If LINE or SMS marketing genuinely need adding, then a notification abstraction (a channel interface plus a unified delivery ledger) earns its keep; before that, any “notification centre” is building a house for an imaginary requirement.

Reflections

Reach is bought, not connected

An engineer’s instinct is “connect the API and you can send notifications” — in reality every channel has its wall: Facebook has policy (one reply per comment, a 24-hour window), email sinks, SMS costs. What’s genuinely scarce isn’t the ability to send, it’s the right to be seen. What “free shipping for a phone number” taught me is that reach should be managed as an asset — it can be purchased (shipping for a number), it depreciates (spam and it’s gone), and it should be rationed (the phone is reserved for a final notice). The first question in designing notifications isn’t “how do I send this”, it’s “what gives me the right to, and why would they look?”

The best infrastructure is the infrastructure you didn’t build

This chapter is about the things that weren’t built from start to finish: no message queue (scan a table), no notification service (two columns), no in-app notifications (email), no chase feature (one query). None of those “nots” is incapacity, they’re judgment — before a requirement is proven, infrastructure is a liability rather than an asset. And what holds those “nots” up is a few “haves” built especially solidly: composable queries, a table at the right granularity of fact, an off-the-shelf Workspace. Subtraction in infrastructure rests on addition in the foundations.

A 99% system, a 1% person

The binding funnel catches 99%, notification delivers 99%, chasing payment ends on a phone call — this system is the same shape everywhere: automation takes the bulk, marks the residue, and hands it to a person. Plenty of engineers treat “1% still needs a human” as a shameful mark and try to automate it to zero; this system treats the 1% as part of the design: giving it a column, a query, and a role responsible for it. A notification’s end point isn’t “sent”, it’s “somebody catches what didn’t arrive” — swap in any system and the sentence still holds: the quality of automation isn’t in how many nines the coverage has, it’s in how gracefully the residue lands.