Cascading Failures and Overload: Don't Let One Server Take Down the Rest

· tech

#sre#reliability

📑 Contents

The first post said the goal of reliability is “keeps working when things go wrong”. But one kind of failure is especially nasty, because it amplifies itself: a small problem knocks over dominoes and takes the whole system down within minutes — the cascading failure. Its most frightening property is that an overloaded system doesn’t slow down linearly; it falls off a cliff.

Cascading failure: how a small fault snowballs into a disaster

The classic script: one machine overloads and goes down, its traffic shifts to the others, so they overload and go down too, one crushing the next like dominoes:

one falls → shifted traffic crushes the next → domino wipeout Server Aoverloads first → down ✗ Server Btakes A's traffic → down ✗ Server Cgets everything → down ✗ and the retry storm pours fuel on the fire: requests fail / slow clients retry total load rises positive feedback: more failure → more retries → more failure (self-amplifying)
The two engines of a cascading failure: traffic shifting makes a fault topple servers like dominoes; the retry storm is positive feedback — failures trigger retries, retries add load, load creates more failures, and a small problem snowballs into a site-wide collapse in minutes

Beyond the retry storm there’s the thundering herd: a cache expires or a service restarts, and a flood of requests hits the backend at the same moment and flattens it. What they share is many requests squeezing through at the same time, in the same direction.

Under overload, protect actively — don’t swallow

Faced with overload, an engineer’s instinct is often “serve as many as we can” — and that’s exactly where the disaster starts: swallowing everything means the queue explodes, resources run out, and then everyone rots together. The right approach is active protection:

❌ swallow it all (passive) overload hits accept everything, queue forever resources exhausted → cliff collapse try to serve all, lose all ✓ protect actively ① Load sheddingdrop some (return 503), save the rest ② Graceful degradationstale cache / switch off extras → good enough ③ Backpressuretell upstream "I'm full" → upstream slows down partial success > total failure Also: a circuit breaker — downstream keeps failing? stop calling it, fail fast, protect both sides
The rule under overload is "partial success > total failure": deliberately drop some (load shedding), fall back to a good-enough degraded version (degradation), or pass "I'm full" upstream (backpressure). Insist on serving everyone, and everyone goes down together

A few moves that keep cascades from happening

Condensed into a practical list:

  • Retry with restraint: cap the count, use exponential backoff + jitter (don’t let everyone retry at once), keep a retry budget (an upper bound on total retries). Unrestrained retries are the biggest accomplice of cascading failure.
  • Circuit breaker: when a downstream keeps failing, stop calling it (fail fast), give it room to breathe and don’t let it drag you down; probe again after a while.
  • Rate limiting / concurrency limits: block the excess at the door instead of letting it in to queue.
  • Capacity planning + load shedding: keep headroom in normal times, shed actively under overload — drop before you collapse, not after.

Reflections

What makes cascading failure terrifying is the positive feedback

Ordinary failures are linear and local: one machine breaks, you’re down one machine. Cascading failure is scary because it has positive feedback — failures trigger retries, retries add load, load makes more failures, self-amplifying into a whirlpool that sucks the whole system in. So when I look at a system I’m especially wary of any loop where “failure makes things worse”: retries without backoff, cache expiry without protection, downstream calls without a circuit breaker — buried positive-feedback bombs, invisible in normal times, and a few minutes from site-wide collapse once lit. Finding and cutting these amplification loops matters far more than firefighting afterwards.

Under overload, “partial success” beats “total failure” by a mile

Load shedding sounds backwards the first time: the system is already struggling, and you deliberately throw requests away? It clicks once you think it through — insist on serving everyone and you lose everyone; drop 10% and you keep 90%. 10% of users getting a crisp 503 is far better than 100% of users timing out together. This trade of “lossy but controlled > lossless but out of control” is exactly the spirit of the error budget and of SLOs: admit you can’t have it all, then pick a point you can hold. A mature system isn’t one that “never refuses”; it’s one that “knows how to refuse, at the right time, with dignity”.

Backpressure: honestly telling upstream “I’m full”

The move I admire most is backpressure. The healthiest systems honestly express their limits — when full, they signal upstream so upstream slows down, instead of pretending they can still swallow and then collapsing together. That’s a kind of humility. I saw the elegant version of the same idea along the Kafka line: when a consumer can’t keep up, it pulls at its own pace rather than being pushed over by the producer — the pull model comes with backpressure built in. Saying “I can’t, please slow down” honestly to whoever is upstream is the most underrated virtue in distributed systems, and in teamwork too: the one who toughs it out and swallows everything is usually the one who drags the whole group down.