Kafka Explained: Partitions, Offsets, Delivery Guarantees and Rebalances

What Kafka actually is

Kafka is not a queue in the RabbitMQ sense. It is a distributed, replicated, append-only log. Producers append records; consumers read forward from a position they control. Reading does not delete anything, which is the property everything else follows from.

• Replay — a new service can start from offset 0 and rebuild its own state from history.
• Multiple independent consumers — analytics, search indexing and billing each read the same topic at their own pace.
• Buffering — a slow downstream falls behind instead of dropping data or crashing the producer.
• Ordering guarantee — strict, but only within a partition.

The partitioned commit log

orders-topic
  P0: [0][1][2][3][4] <- append
  P1: [0][1][2]       <- append
  P2: [0][1][2][3]    <- append

consumer group "billing" (3 consumers)  -> one partition each
consumer group "search"  (1 consumer)   -> all three, own offsets

• The partition is the unit of parallelism. Consumers in a group can never exceed partition count — extras sit idle.
• Partitions can be increased but never decreased, and increasing them changes key→partition mapping, breaking ordering for existing keys.
• There is no global ordering across partitions. If you need order for an entity, all its events must share a key.

Producers, keys and acks

props.put("acks", "all");                    // wait for all in-sync replicas
props.put("enable.idempotence", true);       // no duplicates on retry
props.put("retries", Integer.MAX_VALUE);
props.put("max.in.flight.requests.per.connection", 5);  // safe with idempotence
props.put("compression.type", "lz4");
props.put("linger.ms", 10);                  // batch for throughput
props.put("batch.size", 65536);

producer.send(new ProducerRecord<>("orders", order.customerId(), payload));

linger.ms is the throughput lever most teams leave at 0. Waiting 5–20 ms to fill a batch dramatically improves compression ratio and broker efficiency for a latency cost most pipelines cannot perceive.

Consumer groups and offsets

A consumer group is a scaling unit: partitions are divided among its members, and the group commits its position in the special __consumer_offsets topic. Different groups are fully independent readers of the same data.

props.put("enable.auto.commit", false);
props.put("max.poll.records", 200);
props.put("max.poll.interval.ms", 300000);

while (running) {
  var records = consumer.poll(Duration.ofMillis(500));
  for (var r : records) process(r);       // must be idempotent
  consumer.commitSync();                  // commit AFTER processing
}

• Commit before processing → at-most-once (crash = lost message).
• Commit after processing → at-least-once (crash = redelivery). This is the correct default.
• Consumer lag — log end offset minus committed offset — is the single most important metric you can alarm on.

Rebalances

When a member joins, leaves, or misses max.poll.interval.ms, partitions are reassigned. With the classic eager protocol the whole group stops consuming during the reassignment. Slow processing that overruns the poll interval causes a rebalance, which slows everything further — a rebalance storm. Fixes: use CooperativeStickyAssignor, cap max.poll.records, move slow work off the poll thread, and use static group membership so rolling restarts do not trigger reassignment.

Replication, ISR and durability

• Each partition has one leader and N−1 followers; all reads and writes go to the leader.
• The ISR (in-sync replicas) are followers caught up within replica.lag.time.max.ms.
• Consumers only see records up to the high watermark — the offset replicated to all ISR members.
• The durable production setting is replication.factor=3 + min.insync.replicas=2 + acks=all: you tolerate one broker loss and reject writes rather than accept unreplicated ones.
• Leave unclean.leader.election.enable=false. Enabling it trades data loss for availability.

Modern clusters run KRaft — Kafka's own Raft-based metadata quorum — instead of ZooKeeper: fewer moving parts, faster failover, far higher partition ceilings.

Delivery guarantees

producer.initTransactions();
producer.beginTransaction();
producer.send(outputRecord);
producer.sendOffsetsToTransaction(offsets, groupMetadata);
producer.commitTransaction();

Exactly-once holds inside Kafka. The moment you write to an external database or call a payment API, you are back to at-least-once and you need idempotency keys on your side. In practice: design idempotent consumers and stop chasing exactly-once.

Retention and log compaction

• Time/size retention — delete segments older than retention.ms or beyond retention.bytes. Good for event streams.
• Log compaction — keep the latest value per key forever. The topic becomes a replayable snapshot of current state; ideal for CDC and config/state topics.
• Deletion in a compacted topic is a tombstone: a record with a null value.

Why it is so fast

1. Sequential disk I/O. Appending to a log is near-sequential, which even on spinning disks beats random access by orders of magnitude.
2. Page cache, not a JVM heap cache. The OS does the caching; recent data is served from RAM.
3. Zero-copy (sendfile) sends bytes from page cache to socket without user-space copies.
4. Batching and compression across records amortise per-message overhead end to end.
5. Dumb broker, smart consumer. The broker tracks no per-message state — consumers own their offsets.

Operational traps

• Hot partitions from a skewed key. Watch per-partition throughput, not just topic totals.
• Too many partitions. Each costs file handles, memory and leader-election time; tens of thousands per broker is a rebalance disaster.
• Poison messages blocking a partition forever. Always have a dead-letter topic and a retry budget.
• Unbounded lag alarms missing. Lag is your early warning for every downstream problem.
• Schema drift. Use a schema registry with Avro/Protobuf and enforce backward compatibility, or consumers break on the next producer deploy.
• Consumers doing slow I/O in the poll loop, which triggers the rebalance storm described above.

Kafka vs a message queue

Interview framing

"Order events go to a Kafka topic keyed by customer ID, so all events for one customer land in one partition and stay ordered, while partition count gives us parallelism across customers. Producers run acks=all with idempotence, and topics are replication factor 3 with min.insync.replicas 2, so we survive a broker loss without accepting unreplicated writes. Consumers commit offsets after processing — at-least-once — so every handler is idempotent on an event ID. Billing, search indexing and analytics are separate consumer groups reading the same log at their own pace, and consumer lag is our primary alert. Poison messages go to a DLQ so one bad record cannot stall a partition."