A short Kafka course

Kafka: partitions and consumer groups

Kafka guarantees ordering only inside a partition, and gives each partition to exactly one consumer in a group. Those two sentences sound simple, but between them they explain two things Kafka users hit and struggle to account for: why adding partitions breaks the ordering of a key, and why adding consumers sometimes does nothing at all. This course computes the answers with the algorithms Kafka actually runs, then checks them against measurements on Apache Kafka 4.3.1.

Start reading → Open the lab

Two lessons, two numbers worth keeping Growing a topic from 3 to 6 partitions moved 16 of 32 keys, and Kafka does not move the old data with them. And with the default assignment strategy, a group of ten consumers on a 3-partition topic leaves seven idle — which anyone would predict — but the same strategy over three topics of three partitions each also leaves only three consumers working, where roundrobin gives nine consumers one partition apiece.

The path

Each lesson is a 12–15 minute read with a lab that runs in the browser.

Lesson 1

Which partition a key lands on

The producer's murmur2 formula, checked against 256 key-to-partition pairs. With a lab for your own keys and the cost of growing a topic.

Lesson 2

Who reads what

Four assignment strategies measured on live groups. With a lab for your own topics, partition counts and consumer counts.

How it was measured

The rig is a single Kafka broker in KRaft mode inside Docker, plus shell scripts copied into the container and run with docker exec. No client library sits in between: everything goes through the tools that ship with Kafka.

docker run -d --name kflab -e KAFKA_NODE_ID=1 … apache/kafka:latest
docker cp measure-partitions.sh kflab:/tmp/
docker exec kflab sh /tmp/measure-partitions.sh

The key-to-partition mapping is read with kafka-console-consumer.sh and --property print.partition=true; group assignments are read with kafka-consumer-groups.sh --describe --members --verbose. Both JavaScript engines in this course were checked back against those measurements before anything was written.

One thing only a second run revealed The first measurement showed that the three consumers receiving partitions under range were c1, c2, c3; a rerun with the same names but the reverse join order produced a different three. So the size of each share is predictable while which consumer receives it is not. Lesson 2 says so rather than hiding it.