Master-Replica Replication: Read/Write Splitting and the Oddities of Replication Lag

· tech

#redis#distributed-systems

📑 Contents

However fast one Redis is, it has ceilings on memory and traffic, and when it dies the data hangs in mid-air. The first foundation on the road to high availability is replication: one master handles writes, several replicas each hold a copy and share the read traffic. It’s also the underlying prototype for each shard of Cluster later, and for Sentinel’s automatic failover. Understand it and you’ve actually understood the common skeleton of every replication system.

Master-replica topology: one writes, many read

One master writes, many replicas read applicationwrite → masterread → replicas Masterread/write · sole write pointonly one write Replica 1read-only Replica 2read-only async replication readreads spread over the replicas → scale reads horizontally
The Master is the sole write point (every write goes through it); Replicas can be many, read-only by default, spreading read traffic out to scale. And the master syncing data to the replicas is asynchronous — it replies to the client without waiting for replica acknowledgement. That "asynchronous" buys write speed, but plants the oddity in the next section

Setup is simple; one line on the replica hooks it up:

REPLICAOF 10.0.0.1 6379   # make this instance a replica of 10.0.0.1:6379 (formerly SLAVEOF)
INFO replication          # role:master/slave, connected_slaves, each side's offset and lag

The price of asynchronous replication, and that oddity

“Asynchronous” is the key to every master-replica oddity: the master finishes the write and doesn’t wait for replicas; it tells you “success” right away. That’s fast, but it has two consequences — first, if the master dies suddenly, writes that haven’t reached a replica are simply lost; second, replicas always lag the master by half a beat, which produces this classic scene:

Write, then read the replica right away → stale value Master ① write x=1 → OK Replica ② read x → still old 0 ✗ ③ replication catches up → x=1 replicating… lag window time → need "read what you just wrote" → send that read to master, or WAIT, or tolerate staleness
You write x=1 on the master, get a success reply, and immediately read x from the replica — and get the stale value, because replication hasn't caught up. This is read-your-writes failing: not a bug, but the physical inevitability of asynchronous replication. The fix isn't to eliminate the lag (you can't); it's to classify your reads: critical reads that need "see what I just wrote" go to the master; reads that can tolerate slight staleness can safely go to replicas. That's exactly the replication consistency levels DDIA describes

For stronger guarantees, Redis gives half a toolkit: WAIT 1 100 makes the write block until at least 1 replica acknowledges (or times out); paired with min-replicas-to-write, you can demand “refuse writes unless enough replicas are keeping up”. But these all trade latency for safety; they aren’t free — in essence you’re picking a point between the speed of asynchronous and the steadiness of synchronous.

Reconnecting without starting over: PSYNC partial resync

The connection between replica and master occasionally drops (network jitter). In early versions, a drop meant starting from scratch — the master saved an RDB and transferred the whole thing to the replica, very painful on big instances. Today’s PSYNC is much smarter: the master keeps a replication backlog buffer, and the replica remembers the offset it replicated up to. On reconnect:

  • A short drop (the missing data is still in the backlog) → partial resync: the master sends only the small slice after that offset.
  • Too long a drop / first connection / offsets don’t match (FULLRESYNC) → only then a full sync: transfer the whole RDB.

So that master_repl_offset you see in INFO replication is the yardstick of master-replica progress; the difference between the two sides’ offsets is the live replication lag.

Reflections

Asynchronous replication is Redis’s “be fast” character extended to the replication layer

Redis choosing asynchronous replication (rather than waiting for every replica to acknowledge before replying) is the same character as its persistence not fsyncing always by default: it puts “fast” ahead of “zero loss”. That fits its positioning as a hot data layer — in most scenarios, a few milliseconds of replication lag and losing the last few writes on failure are acceptable prices for its signature low latency. But it’s also a reminder: a system’s defaults hide its values. Before using Redis, you have to agree with its “speed first” stance; if your data can’t lose a single entry, either patch with WAIT or don’t treat it as the source of truth in the first place. A tool’s character has to match your needs; you can’t force it.

Replication lag isn’t a bug, it’s physics; the engineer’s job is “classifying reads”

The first time you hit “wrote it, can’t read it right away”, you think Redis is broken; in fact it’s a physical law of distribution — as long as replication is asynchronous and data has to cross a distance, there is a lag window. Once that clicked, my way of handling it changed completely: stop dreaming of eliminating the lag, and classify each kind of read — which ones “must see what I just wrote” (viewing an order right after placing it, logging in right after changing a password), send those to the master; which ones “slightly stale is fine” (viewing a leaderboard, browsing a product list), send those confidently to replicas for scale. This “route reads by consistency requirement” thinking is the most practical lesson DDIA gave me, and it applies to every system with read/write splitting, not just Redis.

Master-replica is the “prototype skeleton” of every high-availability system

Writing this post made me surer of one thing: the skeleton of Redis master-replica — one primary copy + several replicas + asynchronous sync + a lag trade-off — is very nearly the universal prototype of every replication system. Kafka’s partition + ISR, Postgres primary/standby, MySQL binlog replication, even etcd‘s Raft (just synchronous and majority-based) — same skeleton, differing only in a few knobs: “synchronous or asynchronous, majority or not, who can be the write point”. So I never learn a system’s replication in isolation; I hang it back on this common skeleton — see what it chose on each of those knobs and you understand its trade-offs. Learn Redis’s simplest master-replica pair thoroughly, and Kafka’s and the databases’ replication are variations on the same story. Which is why I say: understand this, and you understand far more than Redis.