Alerting and On-Call: When to Wake Someone Up

· tech

#sre#incident

📑 Contents

The previous post ended on a line: when should you wake someone up? This post answers it. It’s really two things: alerting (what should fire, and to whom) and on-call (how the person who gets called carries it sustainably). The core idea is one sentence: the purpose of an alert isn’t to “notify”, it’s “someone needs to act now”. Hold on to that and a pile of alert-design questions get a yardstick.

Three tiers of alerts: not everything deserves to wake someone

The most common mistake is making “every anomaly” an alert that wakes people. The right approach is to sort by “does it need a person, and how urgently” into three tiers:

An alert comes in: does it need a person? How urgently? Page a person must intervene "now", or users are being affected → wake someone Ticket a person needs to handle it, but it isn't urgent → working hours Log nobody needs to look; keep it for reference / later analysis → disturb nobody Stuff the non-urgent into Page → alert fatigue, crying wolf; the real incident gets ignored
Only "a person must act immediately" deserves a Page. Fail to tier, wake people for everything, and the result is alert fatigue — people numbed by noise, and the genuine emergency drowned out

Alert fatigue is the most real enemy here: when nine out of ten pages at 3am turn out to be “didn’t actually matter”, people start ignoring them, muting notifications, or responding ever more slowly. So the quality of alerts matters far more than the quantity — every useless Page you cut makes the remaining Pages taken more seriously.

Good alerts bind to “symptom + burn rate”, not “cause”

So what kind of alert deserves to be a Page? Two principles. First, bind to symptoms, not causes (continuing the previous post): users care that “the page won’t load”, not that “some CPU is high” — high CPU doesn’t necessarily mean anyone is affected, and waking someone for it is often a false alarm. And every Page should be actionable (there’s a clear thing to do) and novel (not the same noise every day).

Second, and the key move of modern SRE: tie alerts to the error budget‘s “burn rate”. Budget is being spent either way; how fast decides how urgent:

remaining error budget time → 100% threshold burning fast: gone in hours → Page (act now) burning slowly: lasts to month end → Ticket (handle calmly) Bind alerts to "how fast the error budget burns", not to causes like CPU
Same error budget being spent, but the speed decides the urgency: gone within hours → Page immediately; a slow leak that lasts to month end → open a Ticket and handle it calmly. That's the spirit of "multi-window, multi-burn-rate" alerting

On-call: the goal is to stop the bleeding, not to be a hero

Once alerts are set, someone has to answer them. On-call is the responsibility of “carrying the pager, ready to step in at any time”. A healthy on-call has a few requirements: fair rotation (don’t let one person carry it), a cap on Pages per shift (exceeding it means the system or the alerts have a problem, and you fix that rather than gritting your teeth), compensation, and a clear escalation path (you know who to call when you can’t handle it).

But the most important thing is mindset: when paged, the goal is to mitigate fast, not to find and fix the root cause on the spot. At 3am, with a foggy head and high pressure, the right moves are roll back, shift traffic, restart — make users well first, and leave the root cause for daytime, when you’re awake and can investigate properly. Trust the runbook / playbook, follow the mitigation steps, and don’t rely on heroics in the moment — because heroic firefighting doesn’t scale and isn’t sustainable: you save the day today, and tomorrow when you’re on leave it blows up.

Reflections

Alert fatigue is the engineering version of “crying wolf”

What resonated most in this chapter is that it frames a psychological phenomenon as an engineering problem: alert on everything and you’ve alerted on nothing. Once people are numbed by noise, the genuine emergency gets ignored — that’s crying wolf. So the first question I ask of an alerting system now isn’t “is it complete enough” but “how much of this doesn’t actually need attention”. Every useless Page cut buys back attention for the Page that truly matters. It’s in line with post 3’s eliminating toil and my usual “less is more”: an alert’s value lies in precision, not volume.

On-call’s goal is to stop the bleeding, not to be the hero on the spot

I’ve seen (and been) the on-call who gets woken up and insists on digging out and fixing the root cause at 3am — working until dawn, and possibly shipping a fresh mess because the head wasn’t clear. This chapter corrected me: on-call’s first duty is making users well (mitigation), not satisfying your own urge to “solve the puzzle”. Roll back, shift traffic, restart — the “inelegant but effective” moves matter far more at 3am than “finding the truth”; the truth can wait for daylight. And trust the runbook — write the mitigation steps into a manual anyone can follow rather than relying on one hero’s reflexes; that’s operations that scale and let you take leave with peace of mind.

A healthy on-call is a loop that “improves itself”

One last takeaway: a good on-call system makes Pages fewer and fewer, rather than making people more and more tired. The key is that after every page, you seriously ask two questions: can this be automated away (toil)? Can it be fixed at the root (postmortem, next post)? If your on-call handles the same batch of nonsense every week, that isn’t simply “on-call is hard”; it’s nobody running the improvement loop. So I treat on-call as a signal source: where it keeps hurting is exactly where the system most needs investment — woken by the same thing a second time, fix it at the root, rather than resigning yourself to a third.