The Soul of Redis: Five Core Data Structures + Advanced Weapons

· tech

#redis#data-structures

📑 Contents

The previous post said the soul of Redis is “data structures” — so this one opens the toolbox. Ninety percent of using Redis well is picking the right structure: pick right and a leaderboard is three commands; pick wrong and you’ll grind out on the application side with a pile of GET/SET what the server could do in one line. First the five core structures, and each one’s signature use:

The five core structures

Five core structures: pick the right one, the problem is half solved Stringbytes / number 42 cache · counter (INCR) · distributed lock (SET NX) Listordered, both ends queue (LPUSH / RPOP) · latest N items (LPUSH+LTRIM) Hashfield → value name: Aidanage: 30 store objects · change one field without moving the whole thing Setunordered, deduplicated dedup · tags · intersection (mutual friends, SINTER) Sorted Setscored members, kept sorted a:1b:2c:3 leaderboard · range queries · delayed queue (score = time)
Five structures are the answers to five classes of problem: to count, use String (INCR adds one atomically); for first in, first out, a List; to store an object's fields, a Hash; to deduplicate or intersect, a Set; to sort, rank, or take ranges, a Sorted Set. The trick to choosing is to ask first "what operation do I want on this data" — a structure's essence is making the operation you do most into O(1) or O(log N)

The one most worth spending time on is the Sorted Set (ZSet), the jewel in Redis’s crown: every member carries a score, and Redis keeps them sorted at all times. That one property grows into several killer uses — leaderboards (ZADD to score, ZREVRANGE to fetch the board) are the most intuitive; but set the score to a timestamp and it instantly becomes a delayed queue (ZRANGEBYSCORE fetches the tasks “due for processing”); set the score to a page cursor and you get stable range pagination. Same structure, different meaning of the score, a completely different weapon.

Four advanced weapons

Beyond the core five, Redis has a few advanced structures built for specific problems, and their shared philosophy is — trade a little precision or flexibility for a huge gain in space or speed:

Four advanced weapons: trade a little precision or flexibility for space or speed Bitmapone bit per user → check-ins, active-user statsspace: 100M users ≈ 12MB HyperLogLogapproximate cardinality (count distinct)fixed 12KB for 100M+ UV, ~0.81% error Geogeo coordinates (a Sorted Set underneath)nearby users, radius search Streampersistent log + consumer groupsa lightweight Kafka (post 12)
Bitmap expresses boolean state one bit at a time, saving space to the extreme; HyperLogLog gives up a little accuracy (under 1% error) to count hundreds of millions of distinct items in a fixed 12KB; Geo is really a wrapper over a Sorted Set (latitude and longitude encoded into the score); Stream is a persistent message flow, like a lightweight Kafka built into Redis. All of them demonstrate the same trade: give up a little "everything exact, everything possible" in exchange for overwhelming efficiency on one specific problem

HyperLogLog embodies this philosophy best: counting “unique visitors (UV)” with a Set of every user id eats several GB for a hundred million visitors; HLL uses a probabilistic algorithm and a fixed 12KB to estimate a cardinality in the hundreds of millions with under 1% error. For statistics like “roughly a few million UV” that don’t need to be exact, that’s an overwhelming bargain — and a miniature of engineering judgment in general: ask first “does this really need to be exact”; often it doesn’t, and wherever it doesn’t, there’s huge room to optimise.

The rule for choosing a structure: look at the operation first, then pick the structure

The one principle running through all of this: don’t start from “what do I want to store”; start from “what operations do I want to perform on it”. For the same “user score”, if you only need to store and read it back, String/Hash is enough; but the moment an operation like “rank”, “top ten” or “some score range” appears, the answer immediately becomes Sorted Set. Pick the right structure and that operation is a one-line O(log N) command; pick wrong and it’s the disaster of fetching the whole dataset to the application and sorting it yourself. Redis forces you to re-respect something school taught and work makes you forget — the choice of data structure is itself performance design.

redis-cli: the signature commands of the five structures

One set of the most common commands per structure; one look and you catch “what this structure is born to do”:

# String: atomic counter
INCR views:page1                       # +1 in place, no read-add-write
# List: latest feed / queue
LPUSH feed p3; LRANGE feed 0 9         # push at the head, fetch the latest 10
# Hash: an object's fields
HSET user:1 name Aidan age 30; HGETALL user:1
# Set: dedup and intersection
SADD tag:redis u1 u2; SINTER tag:redis tag:db   # people with both tags
# ZSet (the crown): leaderboard
ZADD rank 100 u1 95 u2; ZREVRANGE rank 0 2 WITHSCORES   # Top 3

The commands themselves tell you the structure choice: for a leaderboard, ZREVRANGE is natively a ZSet job; for an intersection (mutual friends, shared tags), SINTER is natively a Set job. Think “which operation am I issuing” first, and the structure surfaces on its own.

Reflections

Picking the right data structure is the watershed of using Redis well

When mentoring new people, one of my favourite indicators is whether they use Redis with only GET/SET. Those who know only those two usually treat Redis as “a faster KV”: fetch and sort for a leaderboard, check an array by hand for dedup; those who use ZSet, Set and Hash write half the code for the same requirement, fast and atomic. The gap isn’t “familiarity with Redis commands”; it’s the habit of “thinking in data structures” — see a requirement, and first ask in your head “which structure does this map to”. It’s also why I think Redis is excellent training for backend engineers: it turns the abstract data structures course into daily practice with immediate performance feedback.

What the advanced structures taught me: ask first “does this really need to be exact”

HyperLogLog was a conceptual shock for me. I used to assume “statistics have to be accurate”, until I understood estimating a hundred million UV in 12KB with under 1% error — for a UV number on a dashboard that people read for trends, does 99% accurate differ from 100%? No, but the cost differs by several orders of magnitude. Since then, before any statistic or any query, I ask one more question: “How exact does this result need to be?” Often the answer is “roughly is fine”, and where “roughly is fine”, the biggest room for optimisation usually hides. Trading a little precision for a huge resource saving is an extremely good deal in engineering, and one that’s often overlooked.

Redis brings the data structures course to life

For many people, the university data structures course meant memorising complexities and handing them back to the teacher after the exam. Redis makes it live: every time you pick a structure, you’re genuinely deciding the system’s performance and behaviour; ZADD is O(log N), the cost of SINTER depends on the smallest set, LINDEX on a big List is O(N) — no longer symbols on an exam paper but the real consequence of “will this drag down the live service”. I even think that to quickly build a solid data-structures foundation in an engineer, having them seriously use Redis for a round beats grinding puzzles — because it puts “the cost of picking the wrong structure” right in front of you, and what has hurt is what you remember.