Pub/Sub vs Stream: Redis's Version of a Messaging System

Redis can serve as a messaging system too, but it has two completely different things for it, and using the wrong one means mysteriously dropped messages or a sledgehammer for a nut: Pub/Sub (broadcast; gone is gone) and Stream (a kept log, like a miniature Kafka). This is the finale of the whole Redis series, and it happens to connect back to Kafka — because Stream is very nearly Kafka’s core concepts condensed into one Redis data structure.

One drops it, one keeps it and can replay

Pub/Sub: broadcast, for real-time notifications where “missing one is fine”

The Pub/Sub model in three sentences: SUBSCRIBE a channel, PUBLISH to the channel, and every subscriber online at that moment receives it immediately. It’s fire-and-forget — the broker doesn’t record who received what, doesn’t resend, doesn’t store. So it’s inherently suited only to real-time broadcasts where “missing one or two doesn’t matter”: online presence, live notifications, cross-node cache-invalidation broadcasts (tell everyone to drop a key).

SUBSCRIBE news          # subscribe (this connection enters subscribe mode)
PUBLISH news "hello"    # another connection publishes; only subscribers online right now receive it

Never use it for task dispatch that “must not be dropped” — during the few seconds a subscriber restarts, or a network hiccup, the messages from that window are gone forever, and you won’t even know they were dropped.

Stream: a kept log + consumer groups, like a lightweight Kafka

For “must not drop, replayable, several workers sharing the load”, use a Stream. It’s an append-only log: XADD writes, every entry gets an increasing ID; more importantly it has consumer groups, whose behaviour is very nearly a copy of Kafka’s:

XADD orders * item book qty 2                 # write one entry (* = auto-generate an increasing ID)
XGROUP CREATE orders workers 0                 # create a group starting from the beginning
XREADGROUP GROUP workers c1 COUNT 10 STREAMS orders >   # c1 fetches new messages
XACK orders workers 1699-0                      # done processing, acknowledge this one (else it stays in the PEL)
XPENDING orders workers                         # see what's been "taken but not acked"

So when should you go straight to Kafka?

If Stream is this much like Kafka, what’s Kafka for? The line is scale and positioning:

• Use Redis Stream: a lightweight message / task queue, volume not outrageous, short retention, you already have Redis and don’t want to stand up a whole Kafka for one queue.
• Go straight to Kafka: very high throughput, long retention (TB-scale, days to weeks, replayable for recomputation), massive fan-out (dozens of consumer groups each reading a copy), the ecosystem (Kafka Connect, Streams), harder durability guarantees. Redis Stream’s data lives in memory (persisted via RDB/AOF, and usually truncated with MAXLEN); it was never designed to “retain vast history”.

In one sentence: Redis Stream is “a handy lightweight queue”, not “a Kafka replacement”. While the need is small, don’t shoulder a Kafka for one queue; but when you truly need what Kafka offers, forcing Stream to cope is rebuilding a crippled Kafka.

Reflections

”Gone is gone” or “kept on the books” — ask this first

The choice between Pub/Sub and Stream comes down to one question: if one message is missed, can you bear it? If yes (real-time notifications, presence broadcasts), Pub/Sub is light and simple; if no (orders, debits, task dispatch), you need something that stores, replays and acks — Stream. It’s the same thinking as the “kept vs taken away” line in my RabbitMQ post and Kafka’s “log vs queue” — before choosing a messaging solution, think through “what happens if one is missed”, and the answer picks the tool for you. I’ve seen too many incidents whose root cause was sending “absolutely can’t be dropped” messages through something fire-and-forget, while everyone kept thinking something else was broken.

Good abstractions “grow into the same shape”

One thing about Redis Stream struck me: it’s very nearly Kafka’s core — append log, offset, consumer group, ack, pending — condensed into one Redis data structure. Two teams, two completely different implementations, and the skeletons they grew are strikingly alike. Which shows those concepts aren’t Kafka’s property; they’re the natural solution to the problem of “reliable durable messaging” — anyone who works at it seriously grows the same log + offset + group + ack shape. It’s also why learning gets cheaper for me over time: understand one good abstraction and you understand a whole batch. Learn Kafka’s consumer groups thoroughly, and Redis Stream and other messaging systems are the same story in a different skin.

Closing: every face of Redis proves “it’s not just a cache”

With this post, the whole Redis series comes full circle. Looking back over the road — data structures, the single thread, persistence, expiration and eviction, the three cache disasters, distributed locks, replication, Sentinel, Cluster, transactions and Lua, and now messaging — every face has been proving the opening line: Redis is an in-memory data structure server, and “cache” is just its most famous use. It can be a lock, a queue, a leaderboard, a counter, a lightweight message log, because it gives you a set of fast, atomic data structures, and the rest is how you combine them. Which is also the one line I most want to leave you with after the whole series: don’t box Redis into the little “cache” compartment — once you start treating it as “a data structure server on call at your elbow”, a lot of problems that used to need big machinery to solve suddenly become light and simple.