The Three Cache Disasters: Penetration, Breakdown, Avalanche, and the Right Fixes

· tech

#redis#cache

📑 Contents

Using Redis as a cache, the classic pattern is cache-aside: a read checks the cache first, returns on a hit, and only on a miss queries the database and writes the result back into the cache. It works fine day to day — until, in certain situations, a flood of requests bypasses the cache and flattens the database behind it. That class of breach has three famous names: penetration, breakdown, avalanche. They sound alike and are often confused, but they’re three different failures, each with its own cure.

The three siblings look different: penetration / breakdown / avalanche

First, the differences — understand where each one “breaches”, and you won’t apply the wrong cure:

Three breaches, three different shapes Penetration query a "nonexistent" key not in cache, not in DB → cache can never block it every one hits the DB Breakdown one hot key ⏰ just expired a burst of concurrent misses → all rush to rebuild from DB concentrated on one point Avalanche many keys ⏰ expire together (or Redis goes down entirely) → misses across the board DB collapses → cascade in common: requests bypass the cache and flatten the DB; the difference is the breach — a nonexistent key (penetration) · one hot spot (breakdown) · a whole swath (avalanche)
The difference in one sentence: penetration queries data that "doesn't exist", in neither cache nor DB, so every single request reaches the DB; breakdown is the instant "one hot key" expires, when a burst of concurrent requests all miss and concentrate on that one point; avalanche is "a whole swath of keys" failing at once (usually because they all got the same TTL, or the whole Redis went down), causing a wide collapse. Work out whether it's "nonexistent / one hot spot / a whole swath" and you know which cure to use

Match the cure to the breach: three breaches, three fixes

With the differences clear, the cures follow naturally — each breach maps to a way of “keeping failures from concentrating”:

Match the cure to the breach: breach → fix Penetration cache the empty result too (write the miss into cache with a short TTL)+ Bloom filter: intercept keys that "definitely don't exist" up front Breakdown mutex: let only "one" request rebuild from DB; the rest wait for the refill(or logical expiry + background refresh for hot keys, no real TTL) Avalanche add random jitter to TTLs to spread out expiry times+ high availability / degradation: don't let Redis be a single point that all dies the shared spirit: don't let failures concentrate — block · converge · spread
Penetration: "cache the empty result too", so queries for nonexistent data are blocked at the cache as well (with a short TTL to avoid stale data), and for more rigour add a Bloom filter at the very front to intercept keys that definitely don't exist; breakdown: a mutex, so when a hot key expires only one request rebuilds it and the rest wait quietly, instead of a thousand arrows at once; avalanche: random TTLs to stagger expiry so everyone doesn't expire in the same second — the same idea as the jitter reliable cron uses against the midnight thundering herd

The three cures use different moves, but at heart they’re one sentence: don’t let failures concentrate at the same time and the same point. Caching empty results is “block what can’t be hit at the door”, the mutex is “converge the concurrency into one”, random TTLs are “spread out what happens simultaneously” — block, converge, spread, matching the three kinds of concentration. As for the mutex breakdown needs, “how to correctly grab a lock in a distributed setting” is a big topic of its own, saved for the next post on distributed locks.

In commands: how the three fixes are issued

In redis-cli, the cures for the three disasters are these few lines:

# cache-aside read: check the cache first, refill only on a miss (always with a TTL)
GET article:42                       # miss → fetch from DB, then refill ↓
SET article:42 "<json>" EX 300       # keep 5 minutes; add some randomness to the TTL → prevents avalanche expiry
# breakdown: a mutex so only "one" request rebuilds the hot spot
SET lock:article:42 1 NX EX 10       # only the one that gets it (OK) queries the DB; others wait and retry
# penetration: cache an empty value for "not found" too (short TTL) to block repeated DB hits
SET article:404 "" EX 30

Three commands for three breaches: a refill with a randomised TTL spreads out avalanches, SET NX as a mutex converges breakdowns, an empty value with a short TTL blocks penetration. The shared sentence is — don’t let a flood of misses hit the DB at the same instant.

Reflections

Three names that sound alike, but at heart three different kinds of concentration

Penetration, breakdown, avalanche — when I was learning, the three Chinese terms left me dizzy; they sound like three ways of saying the same thing. What truly sorted them out for me was realising the difference lies on a single dimension: “why are the requests reaching the DB concentrated?” Penetration because the thing queried doesn’t exist at all, so the cache wall inherently can’t block it (a leak in space); breakdown because one hot spot at the instant of expiry gets everyone piling on (one point in time); avalanche because a whole swath of keys happens to expire together (a wide span of time). Break it into “nonexistent / one point / a whole swath” and the names no longer need memorising; you can derive them from the situation. Faced with a set of easily confused terms, finding the one dimension that distinguishes them is a hundred times more useful than memorising definitions.

The shared spirit of these fixes: eliminate “synchrony”

The biggest takeaway from writing this post was discovering that the three fixes actually solve one deeper problem — synchrony is the amplifier of disasters. Ten thousand requests spread out are digested by the DB with ease; but if they happen simultaneously (expire together, rush the same hot spot together), they crush the system in an instant. So the jitter of random TTLs and the convergence of a mutex are both, at heart, breaking that fatal synchrony. I’ve since seen the pattern everywhere: schedulers adding jitter against the midnight thundering herd, retries with random backoff against retry storms, staggered warm-up at start-up against stampedes — whenever you see “many things doing the same thing at the same time”, be alert that it may be the fuse of the next disaster. Flattening a synchronised spike into a spread-out slope is a move that recurs throughout reliability engineering.

The hard part of caching isn’t the cache; it’s “the moment of the miss”

Using Redis as a cache, everyone’s attention goes to “how fast a hit is”, but where things actually go wrong is always the moment of the miss — the cache didn’t block it and the pressure passes straight to the database behind. All three disasters happen after a miss. So when I design any cache layer now, the first thing I think about isn’t “how high is the hit rate” but “when it misses, when the whole thing goes down, can the DB behind it hold?” A cache whose miss behaviour hasn’t been thought through blocks traffic for you day to day, peace and quiet, but at the same time it feeds a real traffic level the DB behind it can’t bear — and the moment the cache fails, that traffic shows its true face and crushes the DB in an instant (cascading failure). A cache is an accelerator, but don’t forget it’s also a line of defence whose “what happens when it falls” you’ll have to face sooner or later.