The Trouble with Distributed Systems: Unreliable Networks, Untrustworthy Clocks, and Half-Dead Nodes

The previous post ended on a hook: once a system spans several machines, even “locks” and “validation” themselves become unreliable. This chapter is the full reckoning of that “unreliable” — and, I think, the chapter in the whole book most worth reading closely. The core in one sentence: a single machine is deterministic, either all working or all broken; in a distributed world the normal state is “partial failure” — some parts broken, the rest still running, and you often can’t tell which is which. Three unreliabilities, each deeper than the last: the network, clocks, then the node itself.

Unreliable networks: “no response” has four causes, and you can’t tell them apart

You send a request, and no response comes — what happened? The most important diagram in this chapter is the answer to that question: you don’t know, and in principle you can’t.

This “indistinguishability” isn’t sloppy engineering, it’s the nature of an asynchronous network — no mechanism can guarantee a message arrives within any bound. That’s why Sentinel distinguishes subjective down (I think it’s dead = my timeout fired) from objective down (a majority think it’s dead), and why retries must be paired with idempotency — because the request you’re resending may already have succeeded the first time.

Untrustworthy clocks: if you want order, use sequence numbers, not time

The second unreliability is sneakier, because most of the time it looks perfectly normal. First separate the two kinds of clock on a machine, then look at the disaster of misusing them:

Half-dead nodes: you can’t even be sure you’re alive yourself

The third layer is the most philosophical: even a node’s own judgment can’t be trusted. A process can be paused between any two lines of code — a stop-the-world GC, a suspended VM, a page fault that won’t finish — for seconds or even minutes, with no awareness of it. The moment it wakes, it believes it’s still the leader, still holds the lock, but the world has long since moved on. That’s exactly the scene I drew in the distributed lock post: a GC pause exceeds the TTL, and two clients “hold” the lock at once — and the fix (fencing tokens, a monotonically increasing number checked by the resource itself) was covered there too, so I won’t redraw it here.

Put the three layers together and Ch8’s conclusion emerges: no single node’s judgment — including its judgment of itself — can be trusted; so in a distributed system “truth” can only be the result of a majority vote (a quorum). Even if a node believes it’s alive, once a majority declares it dead, it is dead and must step aside. That’s the thread laid all the way from Sentinel’s majority to Cluster’s majority — and “how a majority safely reaches one decision” is exactly the subject of the next chapter, consensus. (As for Byzantine faults, where nodes lie: unless you’re building a blockchain or flight systems, assume nodes are honest but break in your own datacenter — don’t pay design tax for a threat model you’ll never face.)

Reflections

A timeout isn’t knowledge, it’s a decision — once that clicks, retries and idempotency become articles of faith

That “four indistinguishable causes” diagram is the one I’d frame from the whole of DDIA. It punctures a common engineer’s illusion: timeout fired = the other side is dead. No — a timeout firing only means “you decided to stop waiting”; you still don’t know whether the request never arrived, was half done, or was done with the response lost. That “don’t know” derives two practical rules I hold as discipline: first, retries are mandatory, so idempotency isn’t optional (the foundation of the reliability post turns out to be rooted here); second, any single node’s verdict on life or death is only a guess, so to act you need a majority (SDOWN→ODOWN turns out to have its theory here). One chapter gathered three practical habits scattered across Airflow, Redis and SRE into corollaries of the same axiom.

”Want order? Use sequence numbers, not clocks” — data engineering uses this line every day

The scene where LWW drops data on timestamps should sting people in data especially, because we wrestle with its variants daily: event time vs processing time, late events, merge-sorting logs across datacenters. This chapter gave me one unified answer: the wall clock is only for “rough labelling”; wherever correctness depends on order, use a monotonically increasing sequence number — Kafka offsets, database LSNs, fencing tokens, all incarnations of this one principle. Now, designing any pipeline, when I see “use timestamp to decide which is newer” I stop and ask: do these two timestamps come from the same clock? If not, switch to a sequence number, or accept the approximation. The line is cheap, memorable, and blocks a whole class of silent data corruption.

Partial failure isn’t a bug to fix, it’s a worldview to accept

After this chapter my respect for the word “distributed” went a layer deeper: a single machine is the deterministic world of “all good or all broken”; distributed is the probabilistic world of “some part is always half-dead” — and the latter isn’t engineering that’s not good enough, it’s the nature of the thing. That gives me lessons on two levels. Downward, it explains why the K8s reconcile loop and everything in SRE design revolve around “expect failure” — in this worldview reliability isn’t “nothing goes wrong”, it’s the system still converges when something does. Upward, it endorses confirm the pain first once more: every machine you step beyond moves you from the deterministic world into the probabilistic one, and that’s a whole cognitive tax — network, clocks, quorums, fencing, all to be learned. If one machine can carry it, don’t rush to distribute; if you truly must, treat this chapter as the entry ticket and read every word before setting out.