Distributed Locks: From SETNX to Redlock, and That Famous Argument

· tech

#redis#distributed-systems

📑 Contents

When several processes or machines compete for the same resource (only one may decrement stock or run a job at a time), you need a distributed lock. Redis, being fast and atomic, is often used as that lock. But this is a topic that “looks like three lines and turns out to be bottomless” — follow it far enough and you run into a famous academic argument in distributed systems. This post starts from the most naive version, patches it step by step up to Redlock, then faces honestly the question of “is a Redis lock actually safe”.

A correct single-node lock: SET NX PX + Lua release

Start with the most common, “roughly correct” version on a single Redis node. Each of its parts patches a hole the naive version falls into:

Acquiring the lock: one atomic command SET lock:res <token> NX PX 30000 NXset only if the key doesn't exist→ mutual exclusion PX 30000built-in 30 s TTL→ holder dies, auto-release; no deadlock <token>a random value→ marks "this lock is mine" Releasing the lock: verify the owner (Lua, atomic) if GET(key)==token then DEL(key)GET and DEL must be atomic → only delete your own lock DEL without checking token→ deletes someone's renewed lock ✗
Three parts, each patching a hole in the naive version: NX gives mutual exclusion (only the first set succeeds); PX 30000 gives the lock a TTL, so if the holder crashes the lock expires on its own instead of deadlocking forever; <token> is a random value so the release can verify identity. Release must use Lua to wrap "compare the token, delete only if it matches" into one atomic operation — otherwise, if you GET and then, just as you're about to DEL, the lock expires and someone else takes it, you've deleted someone else's lock

In commands it’s these two pieces:

# acquire: NX (exclusion) + PX (TTL) in one go; token is a random value like a UUID
SET lock:order:42 3f9a...e1 NX PX 30000
-- release: GET and compare the token, DEL only if it matches (run the whole thing with EVAL for atomicity)
if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
else
  return 0
end

In the early days people set locks in two steps, SETNX then a separate EXPIRE — never do that: crash between the two and the lock has no TTL and becomes a permanent deadlock. SET ... NX PX fuses “set the lock” and “set the expiry” into one atomic command precisely to plug that hole.

But is that safe? The fatal assumption of a TTL lock

That looks complete. But Martin Kleppmann (author of DDIA) raised a devastating objection: as long as there’s a TTL, the lock cannot guarantee true mutual exclusion. The problem is something nobody escapes — process pauses (a GC’s stop-the-world, OS scheduling, even the machine being suspended):

One long enough GC pause breaks mutual exclusion A acquires lock (TTL 30s) STW GC pause (> 30s) wakes, thinks it holds → writes ✗ TTL up, lock expired B legitimately takes the lock → writes A and B both write → broken time → Fix: fencing token (a monotonically increasing number)the resource accepts only tokens larger than any seen → A's write with the old number (33) is rejected at the resource
The fatal scene: A takes the lock, then suffers a GC pause longer than the TTL; during the pause the lock expires and B legitimately takes the same lock and starts working; A wakes not knowing its lock is long gone and keeps writing — exclusion broken. The key insight: a lock's TTL rests on an assumption about "time", and in a distributed system nobody can guarantee time. The real protection isn't in the lock but at the resource: every acquisition issues a monotonically increasing fencing token, the resource accepts only larger numbers, and A's write with the old number is stopped at the resource's gate

The fencing-token insight matters: a lock can only “try” to exclude; real correctness comes from the resource rejecting stale writes. And a fencing token that increases monotonically needs a reliable counter of its own — which brings you back to the territory of real consensus.

Redlock, and that argument

antirez (Redis’s author) proposed Redlock to make Redis locks more reliable: don’t rely on one Redis; set up N independent masters (usually 5), and to acquire, you must grab the lock on a majority (≥3) within a bounded time, or it doesn’t count; on release, unlock on all nodes. Using “majority” to survive a minority of nodes dying — in spirit the same road as Cluster’s majority failover and consensus algorithms.

Then the argument started. Kleppmann criticised: Redlock depends on timing assumptions like “each node’s clock runs accurately enough, processes don’t pause for long”, and those assumptions don’t hold in real systems (GC pauses, clock drift), so Redlock gives a sense of safety it can’t actually deliver; for correctness you need fencing tokens + a real consensus system. antirez countered: Kleppmann’s attack model is too harsh, Redlock is sufficient for most practical scenarios, and the fencing-token approach carries assumptions of its own. Nobody “won” the debate, but it forced out the most useful distinction we have to this day.

The distinction: do you want an “efficiency lock” or a “correctness lock”

  • Efficiency lock: the lock only exists to avoid duplicated wasted work — recomputing the same cache twice, sending the same email twice; an occasional failure just wastes a little, nothing terrible happens. For this, a single-node Redis SET NX PX is more than enough; you may not even need Redlock.
  • Correctness lock: two parties must absolutely never act at once — deducting money, issuing an invoice, transferring funds. Here a TTL lock’s timing assumptions don’t qualify; you need fencing tokens + real consensus (ZooKeeper, etcd), or simply hand exclusion to the database’s transactions and unique constraints.

Reflections

Things that “look like three lines” are often the deepest

Distributed locks are my number-one teaching example of “the devil is in the details”. Patching from SETNX all the way to Redlock, every step plugs a hole you never thought of at the start: forget the TTL and you deadlock, forget to verify the token and you delete the wrong lock, forget GC pauses and exclusion breaks… Each hole looks “obvious” on its own, but you just don’t see it until you’ve fallen in. The lasting habit this gave me: when the thought “isn’t this simple?” shows up, be more alert, not less — the things treated as three-line trivia are exactly where pits hide in the corners you didn’t look at. Real seniority isn’t being able to write those three lines; it’s knowing what those three lines “still miss”.

Efficiency lock vs correctness lock is the ruler I use most in technical decisions

Kleppmann’s distinction is worth far more than locks alone. It taught me, before using any “best-effort” mechanism, to ask: if it fails occasionally, is that “a little waste” or “someone gets hurt”? If the former (efficiency), happily accept the simple solution’s occasional failure and don’t over-engineer; if the latter (correctness), don’t kid yourself trading timing assumptions for a sense of safety — honestly use real consensus or database transactions. That line freed me from agonising over “is a Redis lock safe”, a question with no absolute answer, and instead I first place “which category does my lock belong to” — classify the problem right, and the tool choice stops being a dilemma.

What that argument taught me: engineering has no silver bullets, only assumptions

Who was right, antirez or Kleppmann? My answer: they weren’t arguing about right and wrong at all; they were arguing about assumptions. Kleppmann assumed a cruel world of GC pauses and drifting clocks; antirez assumed a roughly normal practical environment. Different assumptions, different conclusions — and both assumptions hold in their own scenarios. That changed how I read technical arguments for good: instead of asking “is this solution good”, ask “what assumptions does it rest on, and do they hold in my scenario”. Under every solution that claims to be safe, reliable or fast lies a set of assumptions; laying that set out to look at is a hundred times more useful than listening to anyone’s conclusion. That’s the biggest lesson the small topic of distributed locks taught me.