The Years Without an SRE: A Backend Lead's Production Diary

The last chapter was how the peak hits; this one is about the person fighting it. There was no such job title as SRE on that team — as backend lead, the infrastructure war naturally landed entirely on me. This chapter is the diary of those white-knuckle days: the whole estate on one VM, vague monitoring, four windows open during a stream, and midnight calls to the CTO.

The whole estate: one VM, one database

The position first. The entire platform — traefik, four API processes (Django, doing API and WebSocket together), three Celery containers, plus Redis and RabbitMQ — all crammed onto the same VM. We didn’t run our own database, using Cloud SQL with 8 cores. That’s it: two machines’ worth, holding a stream with up to 13,000 viewers online.

Doing it on as little as possible was a deliberate philosophy, not a lack of budget. That choice cost us later (every fight in #14 was hand-to-hand because of it), but it paid a dividend in discipline: with few resources there’s no escape hatch of “just make it bigger”, and every component has to justify why it deserves a share of a CPU.

How the three Celery containers divide the work is the most designed part of this small position:

• heartbeat: combined with Django for scheduling, so everything that happens on a timer — the rhythm of fetching comments, recomputing sold quantities hourly, scanning for payment chasing — beats from here. As #2 said, Django + Celery + heartbeat was our zero-operations Airflow.
• Fetching comments: exactly one worker. Not thrift, design — a single worker serialises naturally, and the “one fetching job defines the global order by itself” from #3 has this single worker as the physical basis that makes LWW hold. It fetches comments and passes them to the API servers through a Redis group, where the same processes’ WebSocket pushes them to the host dashboard — #9‘s comment waterfall is this line.
• Async tasks: 10 workers over RabbitMQ with acks_late — acknowledge only when done, rather redo than lose. That’s at-least-once semantics, at the cost of tasks having to be idempotent, and #7‘s “fact tables are idempotent by nature” collects interest again here. Slow calls like the invoicing API, taking several seconds each, are absorbed naturally by queuing on RabbitMQ — finish one, then consume the next message, with slow tasks silting up in the queue rather than dragging anyone down. Silts but doesn’t fall, one more time.

Monitoring: Sentry, and our vague selves

Position established; now the embarrassing part: monitoring was essentially all human. That was the limit of our understanding at the time — all we managed was watching errors in Sentry. Sentry’s quota blew up almost immediately after launch, and we learned about sampling afterwards. Nobody on the team really understood monitoring, and I was, at best, watching vaguely.

But Sentry earned one genuine credit, and not in Production — in the demo phase. The system was going out to other departments to try, problem reports came back scattered and fragmentary, and logs inside the VM were sometimes cleaned away and unrecoverable — which is where Sentry earned its keep. That taught me something: logs are volatile, an error tracker keeps the record. Sentry’s real value to us wasn’t “monitoring”, it was persistent memory of errors — the scene had already been swept, and it still remembered what happened.

The blind spot deserves an honest mention too: an error tracker only sees exceptions. Latency creeping up, CPU approaching saturation, a batch silting deeper and deeper — none of those raise, so Sentry stays green while the system may be drowning. In the language of the four golden signals, we could see exactly one of the four: errors. Latency, traffic and saturation were all covered by the humans in the next section.

Watching the stream: four windows

For the first month after launch I sat through every stream, with four windows open:

The first three windows are respectable enough: the dashboard for whether data was arriving, the terminal for each container’s CPU, admin for spot-checking bidding key states. The real alerting system was the fourth — the stream itself. The host-owner saying “the system seems off” was faster and more accurate than any gauge: she stands at the very front of the user experience, and any latency, missed order or page glitch registers with her before it reaches my four windows. #12 said the host is this platform’s notification system; this chapter adds the other half — she’s also the alerting system.

Watching the stream had an exit curve: every stream for the first month; then, once nothing much went wrong, only when the CTO called; and eventually not even that. Retiring human monitoring didn’t come from installing a better dashboard — it came from no longer needing to look. Trust was bought back slowly, one uneventful stream at a time.

Midnight calls, and the language of incidents

Midnight calls didn’t come to me, they went to the CTO — and he was treated that way often. The calls weren’t only incidents, they were also wishes: all kinds of scattered wishes, including the kind where the person wishing couldn’t articulate what they wanted. The most famous one: a Tuesday-midnight call asking for a “shortlist style” feature, needed for Saturday’s stream. We really did rush it out and ship it on time — its full story and its tragicomic ending are saved for the series finale.

Here’s what I only worked out later: what we lacked wasn’t just monitoring, it was a language for incidents.

• No severity levels, so “the system seems off” and “the whole site is down” were the same phone call, and 3am got the same treatment as 3pm.
• No incident entry point, so the only interface for reporting a problem was “call the most senior technical person” — the CTO as human pager and human triage.
• No runbooks, so every call was answered by improvisation.

The person making a wish couldn’t articulate it not because they were unprofessional — it’s that the system never gave them the vocabulary to describe a problem. A user can only say “it seems off” because we never gave them a status page; the boss can only call at midnight because apart from a phone we gave him no severity levels and no entry point. The essence of the alerting and on-call discipline is that it’s a translation machine: turning human unease into system action, and moving the cost of translation from a person into an institution. Without a translation machine the translation work doesn’t disappear — it climbs the org chart until it reaches the most senior technical person, and rings in the middle of the night.

The rebuild: the half of the checklist a pipeline can’t cover

Was there a launch checklist? No, we gritted our teeth and went. The confidence came from infrastructure being in place early: healthy CI/CD, automatic staging deploys — #2 called that a small team’s biggest lever. In hindsight that half of the confidence was real — a repeatable deploy is itself a checklist executing automatically every day, continuously verifying that deployment is correct.

But a pipeline only guarantees the deploy is right; it guarantees nothing about surviving afterwards. The other half of the checklist — capacity, monitoring, alerting — is something a pipeline covers not one item of, and every item on that half-sheet later became a battle: capacity unestimated, so a single process got flattened; monitoring unbuilt, so Sentry stayed vague; alerting ungraded, so the CTO took midnight calls. A rebuild fills in that half:

1. Capacity estimation plus load testing. The opening peak isn’t a black swan — #14 drew its shape: a predictable 200 comments a second. Replay that shape before launch and the single process’s way of dying shows up before going live rather than mid-stream.
2. A minimum monitoring set: the four golden signals plus one business gauge — batch lag (the age of the oldest unprocessed comment). You don’t need the full LGTM stack; one Grafana and a few queries would have done back then. The point isn’t seeing more, it’s turning fear into a number.
3. A language for incidents: three severity levels, one entry point, one page of runbook per level. Make “call the CTO” the last level rather than the only level.
4. Keep watching streams, but change its role: not as monitoring, as product observation. Watching a stream shows you things a dashboard never will — how a host routes around your design, which screen an assistant gets stuck on, which exotic feature nobody actually uses any more.

Would I change the VM? No. One VM plus an 8-core Cloud SQL held 13,000 viewers, proving resources were never the bottleneck; what was missing was protection. Minimalism isn’t the mistake — running blind is.

Reflections

Fear is the most expensive monitoring. The cost of watching streams by hand wasn’t those hours, it was attention and sleep quality — for that first month I was paying for monitoring in fear. The point of a dashboard isn’t letting you know more, it’s permitting you not to look; a monitoring system’s ultimate product is peace of mind. Our exit curve deserves honest criticism too: the correct exit is “the numbers say you can stop looking”, and ours was “we got used to nothing going wrong” — the gap between those two is credit, and the day of the DDoS collected it with interest.

An institution is a translation machine. The line I most want to leave from this chapter: the person making a wish can’t articulate it because we never gave them a language. Severity levels, runbooks, a status page — on the surface these are process, and underneath they’re a translation protocol turning “it seems off” into an actionable signal. Without a protocol the translation doesn’t disappear, it’s just done by a human — always the most senior human, at the hour they should least be awake.

A job title can be absent; the problems never are. There was no SRE role, but not one SRE problem was missing: capacity, monitoring, alerting, on-call — all of them present, just unclaimed. And unclaimed problems grow onto whoever stands nearest. A backend lead doubling as infra isn’t the capable carrying more, it’s a problem finding the closest person to it. After leaving I moved into an EM job, and the first thing I did was take on interim SRE — looking back, the origin is these white-knuckle days.

The system survived, and so did the people. But “alive” isn’t the same as “the books are right” — the stock numbers, the order numbers and the bank’s numbers were drifting quietly in their separate tables. Next chapter: reconciliation.