Lesson 1

Which partition does a key land on

Kafka guarantees ordering inside a partition and nowhere else. That makes “which partition does my key land on” something other than an implementation detail: it decides whether two records about the same customer are read in the order they were written. This lesson computes the answer with the formula the producer actually uses, then checks it against 256 key-to-partition pairs measured on Kafka 4.3.1.

Why the question matters

A topic with P partitions is P independent queues. Consumers read them in parallel and nothing synchronises them: a record on partition 3 may be processed before a record written earlier on partition 0. The only promise is that within one partition, read order equals write order.

So if two records must stay ordered relative to each other — “order KH07 created”, then “order KH07 cancelled” — they have to share a partition, and the only way to arrange that is to give them the same key. The rest of the lesson is about where that key actually goes, and when the promise breaks.

The actual formula

When a record carries a key, the Java producer picks the partition in a single line (BuiltInPartitioner.partitionForKey):

partition = (murmur2(key bytes) & 0x7fffffff) % numPartitions

Three details are worth keeping, because each is a common wrong assumption:

• The hash is murmur2 seeded with 0x9747b28c, not Java's hashCode(). A reimplementation in another language that reaches for a different hash will route the same key elsewhere.
• & 0x7fffffff clears the sign bit; it is not an absolute value. The two differ at exactly Integer.MIN_VALUE, where clearing the sign bit gives 0 while Math.abs overflows and returns the input unchanged.
• The partition count is part of the formula, so changing it changes the mapping. That is the next section.

Three keys from the measured set, step by step:

How it was measured: write 32 keys, KH01 through KH32, into eight topics with 1, 2, 3, 4, 5, 6, 8 and 12 partitions, then read them back printing the partition number.

kafka-console-producer.sh --topic t8 \
  --property parse.key=true --property key.separator=: < keys.txt

kafka-console-consumer.sh --topic t8 --from-beginning \
  --property print.key=true --property print.partition=true
# Partition:6	KH01
# Partition:1	KH02

Growing the partition count: how many keys move

Because the partition count sits inside a modulo, adding partitions pushes some keys elsewhere. Records already written stay where they are — Kafka moves nothing when you run --alter --partitions. The consequence is that for every key that moves, old records live on one partition and new records on another, and ordering between them is no longer guaranteed.

Measured directly: a 3-partition topic, 32 keys written, grown to 6, the same 32 keys written again.

kafka-topics.sh --alter --topic grow --partitions 6
# WARNING: If partitions are increased for a topic that has a key,
# the partition logic or ordering of the messages will be affected

The result: 16 of 32 keys changed partition — exactly half, which is what a modulo predicts when the count doubles. KH03 went from partition 0 to 3, KH05 from 1 to 4, while KH01 and KH07 stayed put.

The whole matrix, counted over those same 32 keys:

Doubling is the cheapest move: 3→6 shifts 16 keys, 4→8 shifts 15, 6→12 shifts 16. Other jumps cost more: 3→8 shifts 28 of 32, and 1→2 shifts 23, because with a single partition everything sits on partition 0 and only the keys that happen to hash back to 0 stay still.

Records without a key do not go round the table

The common belief is that a null key makes the producer cycle through partitions. Measured, it does not: 30 keyless records sent to a 6-partition topic from one producer session all landed on the same partition.

kafka-console-producer.sh --topic nokey < 30-lines.txt

kafka-console-consumer.sh --topic nokey --from-beginning \
  --property print.partition=true | sort | uniq -c
#   30 Partition:5

A second run from a fresh producer session put its 30 records on one partition again — this time partition 3. This is the sticky partitioner, available since Kafka 2.4 and the default since 3.3: it picks a partition and keeps using it until the batch fills or linger.ms expires, because large batches are cheaper than even spreading.

More partitions does not mean more even

A hash spreads evenly in expectation, not over one concrete key set. With exactly the 32 keys above:

At 8 partitions the busiest holds 8 keys against an ideal of 4 — twice — while the quietest holds 1. At 16 partitions three receive nothing; at 24, five do. Raising the partition count raises the ceiling on parallelism, but over a finite key set it also makes the load less even, because each partition holds fewer keys and a single odd key weighs more.

And this is skew in key count only. In a real system record volume differs per key too, so one hot key — a large customer, a chatty device — lands entirely on one partition, and there is no way to split it further while keeping per-key order.

The lab

Type your own keys, choose a partition count, and see where they land. The second box is the partition count after growing: the lab marks the keys that would have to move. The algorithm is a rewrite of BuiltInPartitioner.partitionForKey and matches Kafka 4.3.1 on all 256 measured pairs.

Takeaways

• Same key, same partition — and that is the only way to keep two records ordered.
• The formula is murmur2(key) & 0x7fffffff modulo the partition count, so the partition count is part of the mapping.
• Growing 3 to 6 partitions moved 16 of 32 keys, and Kafka does not move the old data.
• A null key does not round-robin: 30 records from one producer session all landed on one partition.
• With 32 keys over 8 partitions the busiest partition held twice the ideal; at 16 partitions three partitions received no key at all.

The next lesson covers the other half of the story: who reads what, and why nine partitions can still leave seven of ten consumers with nothing to do.