Redis Cluster: How 16384 Slots Shard and Rescale

· tech

#redis#distributed-systems

📑 Contents

The single-thread post said one Redis’s bottleneck is memory and the network. When one machine can’t hold the data, or traffic hits the single-machine ceiling, you shard the data across several machines — that’s Redis Cluster. But its sharding has personality: no consistent hashing; instead the whole key space is cut into a fixed 16384 hash slots. That odd-looking number is actually the key to its scaling being fast and clean.

Which node a key lands on: CRC16 → 16384 slots → node

Where a key lands: CRC16 → 16384 slots → node key: "user:1000" CRC16(key) % 16384= slot 5798 Node Aslot 0 – 5460(+ one replica) Node B ✓slot 5461 – 109225798 is here → B holds it Node Cslot 10923 – 16383(+ one replica) the 16384 fixed slots are a "middle layer"; a node just claims a range of slots so moving data = moving slots; scaling stays clean, no recomputing every key's place
Finding a key's home takes three steps: CRC16(key) % 16384 gives the slot number, then see which node has claimed that slot. The clever part is that layer of fixed 16384 slots — key→slot never changes; what changes is only "which node claims which slots". So when scaling you move whole ranges of slots (with the keys inside), rather than recomputing a pile of keys' destinations as consistent hashing would. It's the same "fixed shard unit" wisdom as Kafka's partitions

Verifying it yourself is easy: CLUSTER KEYSLOT user:1000 returns that key’s slot number. key→slot is a pure function and never changes; slot→node is the half that changes with the cluster.

Clients don’t get lost: MOVED and ASK

Once sharded, how does a client know which node to hit? The answer: the nodes correct you. Hit any node; if that key’s slot isn’t its responsibility, it doesn’t forward for you but replies MOVED <slot> <correct node> — a smart client, on receiving that, caches the whole slot→node map, and from then on hits the right node directly without detours.

A special variant appears during a rescaling migration: while a slot is being moved from an old node to a new one, for “keys already moved” the old node replies ASK (temporary, redirects this one request) rather than MOVED (permanent, update the map). That MOVED / ASK division matches the scaling diagram below.

One restriction: multi-key operations and hash tags

Sharding brings an unavoidable restriction: multi-key operations across slots aren’t allowed. MGET a b c, a MULTI transaction, a Lua script touching several keys — if those keys fall in different slots (most likely different nodes), Redis returns an error outright, because it doesn’t coordinate across nodes.

The fix is the hash tag: wrap a segment of the key in braces {}, and Redis computes the slot from only the braced part. Bind related keys to the same tag and they’re guaranteed to land in the same slot, on the same machine:

# user:1000's several keys, tied to one slot with {1000}
SET {user:1000}:profile "..."
SET {user:1000}:cart    "..."
MGET {user:1000}:profile {user:1000}:cart   # ✓ same slot, multi-key allowed

To operate atomically on a group of keys, tie them together with a hash tag when designing the keys — the habit most worth internalising when programming against Cluster.

The operations: build, scale out, scale in

The main event. Cluster’s day-to-day operations almost all go through the redis-cli --cluster toolset:

# build the cluster: 6 nodes, 1 replica per master (→ 3 masters, 3 replicas)
redis-cli --cluster create \
  10.0.0.1:6379 10.0.0.2:6379 10.0.0.3:6379 \
  10.0.0.4:6379 10.0.0.5:6379 10.0.0.6:6379 \
  --cluster-replicas 1

# inspect (-c makes redis-cli follow MOVED/ASK redirects automatically)
redis-cli -c -p 6379 CLUSTER INFO       # cluster status; look for cluster_state:ok
redis-cli -c -p 6379 CLUSTER NODES      # each node's id, role (master/replica), slots owned
redis-cli -c -p 6379 CLUSTER SLOTS      # slot range → node map
redis-cli -c CLUSTER KEYSLOT user:1000  # which slot a key falls in

# scale out: add a node → move some slots (with their keys) over
redis-cli --cluster add-node 10.0.0.7:6379 10.0.0.1:6379   # join first (a master with 0 slots by default)
redis-cli --cluster reshard  10.0.0.1:6379                 # interactive: how many slots, from whom, to whom
redis-cli --cluster rebalance 10.0.0.1:6379               # or spread slots evenly across all masters automatically

# scale in: move every slot off first, then remove the node (del-node on a node that still owns slots breaks things)
redis-cli --cluster reshard 10.0.0.1:6379 --cluster-from <retiring-node-id> --cluster-to <other-node-id> --cluster-slots 16384 --cluster-yes
redis-cli --cluster del-node 10.0.0.1:6379 <retiring-node-id>

# health check
redis-cli --cluster check 10.0.0.1:6379   # are all slots covered, any inconsistencies
Scaling out = moving slots: add D, hand over a range from each of A/B/C Node A0 – 5460 Node B5461 – 10922 Node C10923 – 16383 Node D (new)claims the moved slots reshard: hand a range of slots (with their keys) from each to D during the move: the old node answers ASK <D> for keys already moved (redirect this once) after: the map updates, and it goes back to a permanent MOVED
Scaling out is in essence moving slots: add a Node D, then reshard to hand a range of slots (with the keys inside) from each existing node to it. Because the shard unit is the fixed slot, you can decide precisely "how many, from whom, to whom"; scaling in is the reverse — reshard all of the retiring node's slots away, then del-node. For a key mid-move, the old node uses ASK to redirect the request temporarily to its new home, and only after the move goes back to MOVED, with no service interruption throughout

Gossip and failover

How do the nodes know each other’s state? Through the gossip protocol — every node continually exchanges “who’s alive, who owns which slots” with the others over the cluster bus, with no central coordinator. When a majority of masters consider some master dead (objectively down), its replica is promoted to take over that range of slots — majority voting showing up in a real system once more. So the recommended number of Cluster masters is odd, and every master should have a replica; otherwise a master disappears along with its slots and the whole cluster refuses service because “some slots have no owner”.

Reflections

16384 fixed slots is a classic victory of “add a layer of indirection”

The first time I saw “16384 slots” it seemed bizarre — why not just hash keys straight to nodes? Once it clicked I was very impressed: it inserts a fixed slot layer between key and node, and that layer of indirection buys the whole cleanliness of rescaling. key→slot never changes; scaling only changes slot→node ownership, and you can say precisely “move these 1000 slots from A to D”, instead of consistent hashing’s “add a node, and a pile of keys get their destinations recomputed as a side effect”. It confirms that old line of computer science — “any problem can be solved by adding a layer of indirection”. The slot is that layer, turning “sharding”, an inherently messy business, into “moving tidy boxes”. When I design any system that has to shard and rebalance now, I first ask: what should my “slot” be? Find that fixed intermediate unit, and the difficulty of rescaling is half solved.

Distribution’s convenience always charges at the boundary

Cluster gives you horizontal scaling, but at the multi-key boundary it posts a clear price: no operating across slots together. It reminds me of a recurring law — the capabilities of distributed systems almost always charge you at some boundary. On a single Redis you can MULTI any pile of keys; on Cluster you have to tie related keys together with hash tags first or forget it. That isn’t a Redis defect; it’s the nature of sharding: the data is cut up and placed on different machines, so “operating together” necessarily costs something. So before adopting any distributed solution now, I ask one question: at which boundary does it collect for the convenience? — multi-key, cross-shard transactions, or cross-region latency? See the toll booth clearly, and the bill won’t shock you after launch.

Confirm you actually need Cluster first

Cold water to finish, as usual. Cluster is powerful, but it complicates a lot: multi-key restrictions, clients must support redirects, operations gain the slot-migration chore. Many scenarios that seem to need Cluster are actually solved by one big enough machine + replicas — a single Redis handling hundreds of thousands of requests a second with tens of GB of memory is routine. You truly need Cluster when the data is too large for one machine’s memory, or write throughput is more than one core can take — those two are hard boundaries; reach them, then adopt. Before that, adding memory, adding replicas for reads, and optimising big keys is usually the better deal. Confirm the pain is “one machine really isn’t enough” before walking into the mountain of sharding — a principle I apply to Redis exactly as to every other heavy weapon.