High Availability: How Sentinel Fails Over Automatically

The previous post‘s replication gave you copies, but left a big hole: when the master dies, nobody takes over automatically. You have to crawl out of bed at night, promote some replica to master by hand, repoint the other replicas at it, then tell every client to switch addresses — a whole scramble. Sentinel is the watchdog that automates that procedure: it detects the master’s death automatically, fails over automatically, and tells clients where the new master is, automatically.

What one automatic failover looks like

The key to the whole flow is the step from ② to ③: a single Sentinel thinking the master is dead is only subjectively down (SDOWN) — its own network may just have hiccuped. Only when a majority of Sentinels agree does it escalate to objectively down (ODOWN) and actually start the failover. A leader is then elected to drive it, the most complete replica is promoted to the new master, and the new address is broadcast. "Detecting" is much harder than "switching" — the hard part is being sure it's really dead, not that the network burped

Sentinel is its own set of independent processes (usually 3 or more, an odd number), running on its own port:

# sentinel.conf: monitor the master named mymaster; the trailing 2 is the quorum
sentinel monitor mymaster 10.0.0.1 6379 2
redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster   # clients ask: who is the master now
redis-cli -p 26379 SENTINEL replicas mymaster                  # see the replicas it manages

Subjective vs objective down: why a majority is required

That “majority” above isn’t arbitrary; it’s the safety core of the whole mechanism. Imagine it weren’t there: any Sentinel that couldn’t reach the master would promote some replica on its own — then under a network partition, each side promotes one, you have two masters (split brain), and the data forks outright. The majority exists to slam that door:

The partition cuts the cluster in half: the minority side (only 1 Sentinel), however much it wants to fail over, can't form a majority and can only wait; the majority side (2 Sentinels) has two votes, confirms ODOWN, and promotes a replica to new master. Because only one side can form a majority at any one time, two masters can never appear at once. It's the same theorem wearing different faces as a consensus algorithm's quorum and Cluster's majority failover — the majority isn't for efficiency; it's so that when the network tears, there is still only one truth

One easily confused detail is worth spelling out: the quorum in the config (the 2 above) only decides “how many Sentinels must agree for ODOWN”; but actually starting the failover and electing the leader to drive it needs authorisation from a majority of all Sentinels. So the total number of Sentinels should be odd and ≥ 3 — both to force out a clear majority when opinions split.

Elect a leader, pick a replica, switch over

Once ODOWN is confirmed, the Sentinels first run a round of Raft-like majority voting to elect a leader, which alone drives this failover (so several Sentinels don’t each switch independently). The leader then picks the most suitable replica to promote — preferring the most complete replication (newest offset, least data lost), then the configured priority. Once promoted, it REPLICAOFs the remaining replicas to the new master and broadcasts a +switch-master event over pub/sub. A Sentinel-aware client doesn’t hard-code the master’s address; it first asks Sentinel “who is the master now”, and on receiving the event automatically reconnects to the new address — the service discovery Sentinel provides as a bonus.

Reflections

What Sentinel taught me: detection is harder than switching, and misjudgment costs more than inaction

The technical act of failover is actually easy — promote a replica, change a few pointers, a few lines and it’s done. Sentinel’s real weight sits entirely on the SDOWN → ODOWN step: how do you “know for sure” the master is really dead, rather than the network jittering or the Sentinel’s own line dropping? That’s the shared difficulty of every automated remediation mechanism — auto-healing, auto-restart, circuit breaking; the actions are easy to write, the judgment is hard. And the cost of misjudging is often greater than doing nothing: the master is actually alive, you misjudge and fail over, and you’ve manufactured an outage that wouldn’t otherwise have happened, or even a double master. So Sentinel puts a safety on detection with “only a majority counts”. It gives me one more measure of respect for any “auto-repair” — first ask how it avoids misjudging, then talk about how well it repairs.

Majority is the immune system of distributed systems

Sentinel hides two layers of majority: judging ODOWN needs one, electing the leader needs another. Both guard against the same thing — a minority of nodes (or the side cut off by a partition) acting on their own and causing split brain. Writing this post, I increasingly see “majority” as a near-universal immune mechanism of the distributed world: consensus algorithms rely on it, Cluster relies on it, etcd / ZooKeeper rely on it, even a relatively plain HA like Sentinel relies on it. Its spirit fits in one sentence: when the nodes disagree, or the network tears everyone in two, let only the side that “can form a majority” act, and the system always has exactly one truth. It’s not for running fast; it’s for not splitting at the most chaotic moment. Understand the majority and you hold the universal key to almost all distributed HA.

Sentinel solves “availability”, not “capacity” — don’t use it for the wrong thing

Finally, the positioning, so you don’t pick the wrong tool. Sentinel competently solves availability: the master dies, someone takes over automatically, no one firefights at night. But it doesn’t solve capacity — the whole setup still has one master carrying all the writes, so the ceiling on data volume and write throughput is the same as a single machine; and failover has a seconds-long gap, and under asynchronous replication may lose the last few writes that never replicated. So its division of labour with Cluster is clear: for “someone takes over when it dies”, use Sentinel (master-replica + automatic failover); only for “one machine can’t hold it or can’t write fast enough” go to Cluster (sharding + master-replica per shard). Recognise that “Sentinel solves availability; Cluster solves availability + capacity”, and you won’t shoulder Cluster’s multi-key restrictions and complexity when you only need the former, nor keep propping up Sentinel when the data has long since outgrown one machine. First see whether you lack “no interruptions” or “enough room”, and the answer surfaces by itself.