Kafka's Core Model: Topic, Partition, Offset and Consumer Group

Part one pinned Kafka’s mental model to a single line: a replayable event log. But how is that log actually laid out, how does it parallelise, and how does a crowd of consumers split the work? This post takes the four core nouns apart — topic, partition, offset, consumer group — and between them they decide throughput, ordering and scale.

Topic and partition: how one log becomes many parallel ones

A topic is a named event stream, a logical category — orders, clicks. But if a topic were literally “one log”, its throughput would be pinned to a single file on a single machine.

So Kafka cuts every topic into several partitions: the thing that is genuinely append-only is the partition, not the topic. “Topic” is just the name for the set of those partitions. The partition is the unit of everything parallel and everything ordered in Kafka — different partitions sit on different brokers and are written and read at the same time, so throughput scales with the partition count.

How a producer decides which partition an event goes to

On write, whether you set a key decides where the event lands — and therefore what ordering you get:

• With a key: Kafka hashes the key and takes it modulo the partition count, so the same key always goes to the same partition. Use order_id as the key and every event for one order is guaranteed to sit in one log, strictly ordered.
• Without a key: events spread evenly across partitions (round-robin / sticky) for maximum throughput, giving up ordering between events.

This is Kafka’s single most consequential design choice: which events are ordered relative to each other is something you decide with the key — it isn’t something Kafka hands you for free.

Offset: an event’s position inside a partition

An offset is the monotonically increasing sequence number of each event within its own partition (0, 1, 2…). It only means anything inside a single partition; comparing offsets across partitions is meaningless.

Back to the bookmark picture from part one: where a consumer has read to is “I’m at offset 5 in P0 and offset 3 in P1”. That position gets committed, so a consumer that restarts picks up where it left off — and when you commit is exactly the source of the “duplicated or dropped” problem the next post is about.

Consumer group: how consumers divide and scale

One consumer can’t keep up — now what? Put several consumers into one consumer group, and Kafka assigns the partitions across its members so they share the load.

There are only two rules, and both matter:

• Within one group, each partition is assigned to exactly one consumer. So consumers in a group split a topic between them — which makes Kafka behave like a horizontally scalable queue.
• Different groups each read everything, independently. Billing has its group, recommendations has its own, and both see every single event — which makes Kafka a broadcast at the same time.

Two corollaries you’ll actually use:

• Your ceiling on parallelism is the partition count. More consumers in a group than partitions and the extras just sit idle — one partition is never fed to two members at once.
• Membership changes trigger a rebalance. A consumer joins or dies and Kafka redistributes the partitions. Consumption pauses briefly during a rebalance, and you feel it when there are many partitions and many members — a cost worth knowing about before you’re operating it.

The ordering guarantee holds inside a partition, and nowhere else

Boiling all of the above into the one line worth remembering: Kafka guarantees strict ordering within a single partition, and guarantees nothing across partitions.

So ordering in Kafka is designed, not default: whichever events you want ordered, give them the same key so they land in the same partition. Global ordering across a whole topic? That means forcing the partition count to 1 — giving up parallelism entirely, and rarely worth it. In practice you take a step back and ask “what’s the unit I actually need ordered?” — usually “one order” or “one user”, not “the whole world”.

Reflection

The partition count is the most important early decision, and the hardest to walk back

The partition count decides three things at once: the ceiling on throughput, the ceiling on consumer parallelism, and the granularity of ordering. And it only goes up — adding partitions later is easy, but it scrambles where existing keys hash to (a key can move to a different partition, and its history no longer joins up). So my habit is to pick a number with room to grow, but not an absurd one, at the start — better slightly generous on day one than discovering in production that you’re pinned and can’t move. It’s one of the few places in Kafka where not thinking it through up front really hurts later.

Choosing a key is designing your ordering and your load distribution

I think the partition key is badly underrated — on the surface it’s just “which log does this row go into”, and in reality it decides ordering and load balance in one stroke. Pick a key that’s too coarse (a constant, say) and everything piles into one partition: a hotspot, parallelism gone. Pick it well (order_id) and you keep “one order stays ordered” while spreading the load evenly. Work out what your unit of ordering is first, then go back and pick the key — the ordering requirement drives the key, not whichever column came to hand.

The consumer group is what lets Kafka be a queue and a broadcast at once

When I was learning this I kept getting stuck on: is Kafka a queue or is it publish-subscribe? The answer, it turns out, is hidden in the consumer group design — split the load inside a group (like a queue), broadcast between groups (like pub/sub). Need one service to consume faster? Add consumers to its group. Need a new team on the same events? Give them a new group. One mechanism, two uses — the prettiest part of Kafka’s model, to my eye.

But ordering and assignment still don’t answer “can it duplicate or drop?”

This post is about how events are laid out and who reads them, and there’s a sharper question it hasn’t touched: when a consumer commits its offset decides whether a crash-and-restart means reprocessing or losing events. That’s the part of Kafka-as-infrastructure that most deserves to be taken seriously — delivery guarantees — and the next post covers at-least-once, exactly-once and replication in one go.