What Is Apache Kafka? From Message Queue to Event Streaming
· tech
📑 Contents
- What Kafka is
- What it solves: from N² direct connections to one hub
- The core mental model: a replayable log, not a queue
- How it differs from a database and from a message queue
- What it’s good at, and what it isn’t
- Reflection
- Treat Kafka as a log of facts, not a pipe for messages
- Its value is decoupling, and decoupling has a price
- It happens to connect to the pipeline I’d been learning
What Kafka is
In one sentence: Apache Kafka is a distributed event streaming platform — it collects the events flowing between systems into a High-throughput, durable, replayable log, so that many producers keep writing and many consumers each read independently. It came out of LinkedIn (2011), became a top-level Apache project, and is now the backbone of almost every real-time data pipeline.
The key realisation: don’t picture Kafka as a traditional message queue (deliver the message, then drop it) — picture it as an event log that keeps being appended to and can be read from the beginning again. That shift is the key to every design decision in it.
What it solves: from N² direct connections to one hub
Once you have a few systems, “who needs whose data” explodes. The order system has to notify inventory, fulfilment, recommendations, risk control, reporting… wire every pair together directly and the number of integrations is on the order of N² — add one new consumer and you go back and change the producer. Fragile, and miserable to maintain.
Kafka collapses that mesh into a hub: producers only write events in; consumers each subscribe to what they want; neither knows the other exists. The systems are decoupled from that point on.
The core mental model: a replayable log, not a queue
A traditional message queue (the RabbitMQ family) works like this: a consumer takes the message, processes it, and it disappears from the queue. Kafka is nothing like that — an event written in is appended to a log that only grows, and it stays there for the whole retention period. Consumers don’t take messages away; each holds a bookmark — an offset (which record it has read up to) — and reads down the log.
That gives you two things a traditional queue can’t:
- Consumers are independent of each other: the reporting team, the risk team and the recommendation team can all read the same log at the same time, each tracking its own offset, without interfering or competing.
- Replay: you shipped a bug, or a new consumer wants the historical data — wind the offset back and read again. The data is still there.
How it differs from a database and from a message queue
| Traditional message queue | Database | Kafka | |
|---|---|---|---|
| Data model | transient messages | queryable current state | append-only event log |
| After reading | the message is gone | data stays, can be changed | stays and can be replayed, but not changed |
| Multiple consumers | they compete | each queries | each reads independently, no interference |
| Primary query | take the next one | arbitrary conditions | read forward by offset |
| Good at | dispatching tasks | the truth of current state | High-throughput event streams, decoupling, replay |
One line to remember it by: a database stores “what the state is now”, Kafka stores “what happened”. They complement each other; neither replaces the other.
What it’s good at, and what it isn’t
- Good at: High-throughput event streams, decoupling systems, sharing one set of events across teams, anything that needs replay, and being the intake for stream processing.
- Not good at: being your primary database (it isn’t designed for arbitrary queries), low-latency request-response interaction, or small message volumes that need complex routing — a traditional queue or a direct API call fits those better.
Reflection
Treat Kafka as a log of facts, not a pipe for messages
The biggest turn for me while learning Kafka was to stop thinking about it in the frame of “message delivery”. Once you see it as an immutable log recording what happened, a lot of the design suddenly makes sense: why the data stays after being read, why several teams can each have their own view, why you can rewind and recompute. The problem with a traditional queue is that a consumed message is gone; Kafka treats an event as a fact worth keeping, not a notification to be used once and discarded. Turn that lens and what it can and can’t do becomes obvious.
Its value is decoupling, and decoupling has a price
Kafka’s most concrete benefit is prising “who produces” fully apart from “who consumes” — add a new consumer and you touch nothing upstream. In a multi-team system that’s worth a great deal. But I also remind myself not to romanticise it: putting a Kafka in the middle means one more piece of infrastructure to operate, monitor, and understand — partitions, offsets and all. Two or three services passing the occasional message are usually better served by a direct API call or a lightweight queue than by carrying a whole Kafka. It was built for the scale where there are many events, many consumers, and a need to replay — the same attitude I took writing about Airflow and Spark: confirm the pain has reached that scale before you bring in the heavy weapons.
It happens to connect to the pipeline I’d been learning
Kafka isn’t a standalone thing; it completes the jigsaw of my recent notes. In the Medallion architecture it’s the most common entry point for pouring real-time events into the Bronze layer; and Spark Structured Streaming is often hanging off the back of Kafka, consuming that log in real time and turning it into Silver and Gold. Compared with Airflow‘s batch scheduling, Kafka represents the other axis — from “run it every hour” to “handle it the moment it happens”. Seeing that axis clearly is what this series goes on to unfold.