Testing for Reliability: Tests Don't Prove the Absence of Bugs, They Let You Move Fast

This post is about something usually filed under “the developers’ business” that is in fact a cornerstone of reliability: testing. The key idea first: reliability isn’t achieved by “not changing” — you have to change (fix bugs, add features, alter config), and every change is a gamble. The point of testing is to turn that gamble into a confident step forward, so you dare to release small and often (which is exactly what the error budget wants: changes you can afford, and can back out of).

The test pyramid: many at the bottom, few at the top

Tests come in layers, and their numbers should form a pyramid — many at the bottom (cheap, fast, stable), few at the top (expensive, slow, brittle):

Green doesn’t mean Production is healthy: finish with a canary

But here’s a truth SRE particularly cares about: all tests green only means “the scenarios you thought to test” passed — real-world traffic, data and timing always contain something you didn’t test. So passing in the test environment isn’t enough; the last line of defence is the canary release: the new version first goes to a small slice of real traffic, you watch the SLIs, and only if all is well do you roll it out fully:

Two more that are often overlooked: configuration needs testing too — many outages come from changing config rather than code, and config often ships untested; and proactive disaster drills — Chaos Monkey style, deliberately injecting faults to test “what happens when it breaks”, not just “what happens when it’s fine”.

Flaky tests are the testing world’s “crying wolf”

One last disease that must be cured: the flaky test — occasionally red, green again on re-run. It’s more toxic than no test at all, because it trains the whole team into the habit of “red light, re-run first, it usually passes”, so genuine red lights get ignored too. It’s the same disease as alert fatigue: noise numbs the signal. My rule is hard: a flaky test gets fixed the same day or removed, never left — left alone, it slowly corrodes the whole team’s trust in “green”.

Reflections

The purpose of tests isn’t “proving there are no bugs”, it’s “letting you move fast”

When I was younger I treated tests as a checkpoint to “prove my code has no problems” — high pressure and frustrating (because you can never finish proving it). Then the mindset shifted: tests can’t prove the absence of bugs (you can’t enumerate everything), but they drastically lower the risk of a change — and once the risk is low, you dare to move forward in frequent small steps. Reliability has never been bought with “change less, don’t touch it”; it’s bought with “dare to verify often”. That’s fully consistent with the error budget and with what I said about K8s rolling updates: “make change reversible, and people dare to move forward often”. Treat tests as an accelerator, not a roadblock, and your relationship with them straightens out.

Green only means “the ones you tested” passed

All tests green feels great, but what it guarantees stops at “the scenarios you originally thought to test”. Real-world traffic distributions, dirty data, odd timings are always outside your tests. So I learned not to treat “tests passed” as “definitely fine”, but as “risk lowered enough to let a small slice of real traffic verify it” — then finish with a canary. The test environment gives you the confidence to bet; the canary lets you bet only a small stake. That mindset of “stake in batches, watch as you go” is far more practical than chasing “test everything watertight before launch” — because the latter simply can’t be done.

Flaky tests corrode the whole team’s judgement

A test that’s occasionally red damages more than itself; it damages the credibility of the whole test suite. Once “red = just re-run” becomes the team’s muscle memory, you’ve effectively muted the alarm system — when something is genuinely wrong, that red light earns only one more unconscious re-run. It’s the same thing as alerting and my constant “signals must be precise”: better one fewer test than one test that lies. Maintaining the “credibility” of tests matters as much as maintaining their “coverage” — a green light nobody believes is no different from no light at all.