Consumer lag is climbing. Yesterday the pipeline was two seconds behind; today it is forty minutes behind and the gap is widening. The instinct is to add consumer instances. Often that does absolutely nothing — and understanding why is the whole question.
What lag actually measures
Lag is per partition: the difference between the partition's latest offset and your group's committed offset for it. Message count, not time. Two consequences people get wrong:
- Lag in messages is not lag in time. 100,000 messages of lag on a topic doing 100/sec is 16 minutes behind; on a topic doing 50,000/sec it is two seconds. Convert lag to time before you decide how bad it is, and alert on the time.
- Aggregate lag hides the cause. Total lag of 400,000 could be 4,000 evenly spread across 100 partitions (you need more throughput) or 399,000 on one partition and almost nothing elsewhere (you have a key-skew problem, and adding consumers will not help). These have completely different fixes.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \ --describe --group order-processor TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID orders 0 1,204,551 1,204,602 51 consumer-1 orders 1 1,198,320 1,198,377 57 consumer-2 orders 2 904,113 1,302,884 398,771 consumer-3 <-- here orders 3 1,201,044 1,201,090 46 consumer-4 One partition holds 99% of the lag. This is skew, not capacity.
Three problems that look the same
| Underlying problem | Signature | What helps | What does nothing |
|---|---|---|---|
| Per-message processing is too slow | Lag rises evenly across all partitions; consumers are busy; CPU or downstream latency high | Optimise the handler, batch downstream writes, add consumers up to partition count | Nothing, until the handler gets faster or you add partitions too |
| Not enough parallelism | Lag even; consumers at 100% CPU; consumer count already equals partition count | Increase partition count, then add consumers | Adding consumers beyond the partition count — they sit idle |
| Skew, stalls or rebalance churn | Lag concentrated on one partition, or consumers repeatedly rejoining | Fix the key distribution, the poison message, or the session/poll config | Scaling anything at all |
Diagnosing per partition
- Per-partition lag, converted to time. Is it even or concentrated? That one answer splits the diagnosis in half.
- Consumer-side metrics.
records-consumed-rate,fetch-latency-avg, and above allprocess-latencyper message. If fetch is fast and processing is slow, it is your handler; if fetch is slow, look at broker health, network, or an oversizedmax.poll.recordscreating long pauses. - Rebalance count. Frequent rebalances mean the group spends its time reassigning partitions rather than consuming. Any non-zero steady-state rebalance rate is a bug.
- Is the offset moving at all? Flat committed offset with rising end offset means stuck, not slow — a poison message, an infinite retry, or a deadlock on a downstream call with no timeout.
- What did the handler start doing? A handler that grew a synchronous HTTP call or an extra database round trip per message will halve throughput without any code looking obviously wrong.
Hot partitions and key skew
Kafka assigns a message to a partition by hashing its key. If keys are skewed, partitions are skewed, and no amount of scaling fixes it because one partition can only be consumed by one consumer.
key = tenant_id → your biggest customer is 60% of traffic key = country → one country dominates key = null → round robin, no ordering guarantee at all key = "order_events" → a constant! every message on ONE partition
Options, each with a cost:
- Choose a finer key.
tenant_idbecomesorder_idif you only need per-order ordering. Usually the right answer — ask what ordering you actually require, which is normally narrower than what you chose. - Salt the hot key.
tenant:42#{0..9}spreads one tenant across ten partitions, giving up strict ordering within that tenant. - Route the whale to its own topic with its own consumer group, so it cannot starve everyone else. This is the bulkhead pattern applied to a stream.
- Decouple ordering from consumption. Consume in parallel, then order at the point where it matters using a version number or sequence check on write.
Note that increasing partition count changes the hash mapping for existing keys, so ordering guarantees break across the resize. Plan it, do not do it during an incident.
Rebalance loops
The most under-diagnosed cause. If a consumer does not call poll() within max.poll.interval.ms, the coordinator assumes it is dead and rebalances. During a rebalance, consumption stops for the whole group. If your handler occasionally takes longer than the interval, you get a loop: process slowly, get kicked, rebalance, restart from the last commit, reprocess, get kicked again. Throughput goes to nearly zero while every consumer looks busy.
# Symptom in logs, repeatedly: # "Member consumer-3 sending LeaveGroup request ... consumer poll timeout has expired" max.poll.records = 500 # 500 records × 300ms each = 150s of work per poll max.poll.interval.ms = 300000 # 5 min — must exceed the time to process one batch # Fix option A: process less per poll max.poll.records = 50 # Fix option B: raise the interval (only if work is genuinely long) max.poll.interval.ms = 900000 # Keep liveness separate from work duration: session.timeout.ms = 45000 # heartbeat thread, unrelated to processing heartbeat.interval.ms = 3000 # And reduce the blast radius of every rebalance: partition.assignment.strategy = CooperativeStickyAssignor
Also check for consumers that die and restart — an OOM every few minutes causes a rebalance every few minutes, and the lag graph looks like a slow consumer.
Poison messages
One message throws on every attempt. Your handler retries forever, or crashes and restarts from the same uncommitted offset. The partition is permanently stuck at one offset while its end offset races away. Fix the pattern, not the message:
- Bound retries per message, with backoff.
- After the bound, publish to a dead-letter topic with the error, offset and payload, then commit past it.
- Alert on dead-letter volume — silent DLQs are lost data.
- Make the handler idempotent so replaying after a crash is safe, per the idempotency pattern.
for (const record of batch) {
try {
await withTimeout(handle(record), 5_000); // never unbounded
} catch (err) {
if (record.attempts < 3) {
await retryTopic.send(record.withIncrementedAttempts());
} else {
await dlq.send({ record, error: String(err), offset: record.offset });
metrics.increment("dlq.messages");
}
}
}
await consumer.commitOffsets(); // always move forwardThe four ways to go faster
- Make the handler cheaper. The highest-leverage fix and the one people skip. Batch database writes for a whole poll instead of one per message; remove synchronous HTTP calls from the hot path; cache lookups. Going from one insert per message to one bulk insert per 500 messages is routinely a 20x improvement.
- Increase partitions, then consumers. Parallelism is capped by partition count. Add partitions first (accepting the key-remapping consequence), then scale consumers to match.
- Parallelise within the consumer. Fetch a batch, process records concurrently across a worker pool, commit only the contiguous prefix that completed. This gains throughput without more partitions, but you lose strict per-partition ordering unless you shard the pool by key.
- Tune fetching. Larger
fetch.min.bytesandmax.partition.fetch.bytesreduce round trips; compression cuts network cost. Worth a modest gain, and only after the handler is efficient.
And two mitigations for the incident itself: if the backlog is historical data you no longer need, seek the group forward to a timestamp (deliberately, with sign-off) to recover real-time processing; and if this pipeline shares infrastructure with a latency-sensitive one, separate them so batch catch-up cannot starve live traffic.
Parallelism versus ordering
| What you need | Design | Max parallelism |
|---|---|---|
| Total ordering across the topic | One partition | 1 — and it will not scale |
| Ordering per entity | Key by entity id | Partition count |
| No ordering requirement | Null key or round robin, parallel workers | Effectively unbounded |
| Ordering per entity, high throughput | Parallel workers sharded by key hash inside the consumer | Worker count, ordering preserved per key |
Most systems over-specify ordering. "Events for one order must be in order" is a real requirement; "all events must be in order" almost never is, and it costs you the entire scalability of the pipeline. For the broader model of partitions, offsets and delivery semantics, see Kafka explained, and for the architectural context event-driven architecture.
How to answer this in an interview
"First, lag per partition converted to time — aggregate lag hides the cause. If lag is even, it's throughput: I check whether processing time per message went up, and whether consumer count already equals partition count. If it does, adding consumers is useless — extra consumers idle — so I either make the handler cheaper (batch the downstream writes, that's usually 10-20x) or add partitions. If lag is concentrated on one partition, it's key skew: a tenant_id key where one tenant is most of the traffic. Then no scaling helps; I'd use a finer key, salt the hot key, or move that tenant to its own topic. If the committed offset isn't moving at all, it's stuck, not slow — a poison message with unbounded retries, or a rebalance loop because processing a poll batch exceeds max.poll.interval.ms. Fix with a smaller max.poll.records, bounded retries and a dead-letter topic. Trade-off: any parallelism beyond one worker per partition costs strict ordering, so I'd first ask what ordering we actually need — usually per entity, not global."
Summary
- Measure lag per partition and convert it to time; aggregate lag hides every cause.
- Even lag means throughput; concentrated lag means key skew; a frozen offset means stuck.
- Partition count is the hard ceiling on consumers in a group — extra consumers idle.
- Batching downstream writes in the handler is usually the biggest single win.
- Rebalance loops come from processing a poll batch for longer than
max.poll.interval.ms. - Bound retries, dead-letter poison messages, and keep handlers idempotent.
- Ask what ordering you truly need — global ordering costs you all your parallelism.