Monitoring: The Four Golden Signals

The previous post said the SLI is the measured reliability number — and those numbers come from monitoring. But where monitoring most easily goes wrong is treating it as “the more numbers collected, the better”, ending up with a hundred charts on the dashboard and no way to find the point when something actually breaks. This post covers Google’s distilled answer: if a service could watch only four metrics, which four — the four golden signals.

Monitoring only has to answer two questions

Let’s be clear about the purpose first. Monitoring isn’t for pretty dashboards; it’s for answering two questions: “is it broken right now?” (real-time detection) and “is it about to break?” (trend warning). Hold on to those two questions and you won’t fall into the trap of “measure everything, see nothing”. And what answers both, while covering nearly all of a user-facing service’s health, is these four signals:

The point of each:

• Latency: how long a request takes to return. One trap — always separate the latency of successes from failures. Failed requests can be very fast (a straight 500) or very slow (stuck until timeout), and mixing them into the successes badly distorts the average.
• Traffic: how busy the system is, usually QPS or transactions per second. It’s the backdrop for understanding the other three — did latency rise because traffic spiked, or because the system has a problem?
• Errors: the proportion of failed requests. Watch for “hidden failures” — a 200 with wrong content; and “policy failures” — responses so slow they count as failures for you.
• Saturation: how full the system is, how close to its limit (CPU, memory, connection pool utilisation). It’s the hardest to measure, yet the best early warning — because it tells you “how long you can hold”, not “it’s already down”.

Don’t look at the mean; the mean lies

The second concept you must build: when looking at latency (or any distribution), don’t look at the mean, look at the distribution — especially p99. The mean is the most deceptive, because it hides the long tail:

That also explains why the previous post‘s SLIs/SLOs almost all use percentiles (“99% of requests < 300ms”) rather than “mean < 300ms”. The mean flatters you; p99 is honest with your users.

Alert on symptoms, and use black-box and white-box together

Two last points, leading into the next post. First, monitoring should watch “symptoms”, not “causes”: what users care about is “the page won’t load” (a symptom), not “CPU is high on some DB” (a cause — which may not have affected anyone). That’s why the golden signals work so well: they’re symptoms by nature. Second, use black-box and white-box monitoring together: black-box hits your service from outside like a user does (“is it actually up right now”), white-box looks at internal metrics from inside (helping you find “why” when something breaks). Black-box catches symptoms, white-box finds causes — use them as a pair. As for “when to wake someone up”, that’s the next post on alerting.

Reflections

The discipline of “only four” beats “measure everything”

Nine times out of ten, the monitoring problems I’ve seen weren’t “measuring too little” but “measuring too much, too messily” — hundreds of charts nobody really understood, and when something broke everyone fished around in a sea of dashboards without finding the point. The value of the four golden signals isn’t which four it lists; it’s that it forces focus: get these four right first, then talk about the rest. It’s the same habit I bring to anything — grab the few that matter most rather than greedily wanting everything. Monitoring maturity is daring to watch only the few signals that truly matter.

The mean is the most deceptive statistic

“Don’t look at the mean, look at p99” applies far beyond monitoring, I think. The very nature of a mean is to flatten differences, and the real problem is usually hiding in the flattened tail — true of latency, of cost, of response time, even of team load. The few extreme values (the 1% of users who waited a second, the handful of oversized requests) are what bite you, and the mean makes them invisible. So whenever I look at any metric now, my reflex is to ask “is this a mean? What does the distribution look like? What about the tail?” — having been reassured by a mean too many times, I learned.

Alert on symptoms, not causes

This principle changed how I design monitoring. I used to be unable to resist setting alerts on every internal metric (CPU, memory, queue length), and got woken at night by a pile of causes that “didn’t actually affect users”. Then it clicked: what should wake a person is a symptom (users are affected); causes are clues you need when diagnosing. Tying alerts to symptoms (golden signals, SLIs) not only removes a pile of false alarms, it aligns “when to worry” with “are users hurting”. Which is exactly what the next post unfolds: alerting — under what conditions to wake someone up.