Kafka Common Errors and Fixes with Practical Examples

Intro

Apache Kafka is a distributed streaming platform that thousands of companies rely on for real-time data pipelines and event-driven architectures. When Kafka fails, the impact cascades quickly: consumers stall, producers time out, and dashboards go stale. This guide collects the most common Kafka errors, their causes, and step-by-step fixes. Each problem includes concrete commands, expected output, and verification steps so you can move from the observed symptom to a confirmed resolution.

This article targets developers, DevOps consultants, and technical startup teams who operate Kafka clusters or build applications on top of them. It connects Kafka error messages, debugging techniques, and troubleshooting patterns to version-specific commands and recovery decisions. The examples assume a Kafka deployment version 2.8 or later, but they call out differences where they matter.

Operational safety is the core principle: observe before changing, limit blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. The article avoids storing real credentials, tokens, private keys, or production identifiers in commands or outputs.

Version and Environment Inventory

Before changing anything, identify the installed Kafka version, deployment topology, and the component that is misbehaving. Kafka errors often look identical across brokers, producers, consumers, ZooKeeper, or KRaft metadata nodes, but the correct fix depends on the component and version. Use version-appropriate commands from the official documentation and capture read-only state before any intervention.

Gather the baseline

On every broker or client host, run:

kafka-topics.sh --version

Expected output (example):

3.6.1 (Commit:... )

If the command is not found, the Kafka binaries are not in PATH. Locate the Kafka installation directory, typically /opt/kafka or /usr/local/kafka, and use the full path:

/opt/kafka/bin/kafka-topics.sh --version

Check the broker's server.properties for the advertised listeners and log directories:

grep -E '^(broker.id|advertised.listeners|log.dirs|zookeeper.connect|process.roles)' /opt/kafka/config/server.properties

Example output:

broker.id=1
advertised.listeners=PLAINTEXT://broker1:9092
log.dirs=/var/lib/kafka/data
zookeeper.connect=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181

For KRaft clusters (Kafka 3.3+), inspect metadata properties instead:

grep -E '^(process.roles|node.id|controller.quorum.voters|advertised.listeners)' /opt/kafka/config/server.properties

Example output:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@kafka1:9093
advertised.listeners=PLAINTEXT://kafka1:9092

Record the Kafka version, topology (broker IDs, ZooKeeper ensemble, or KRaft quorum), and the component under investigation. Save this baseline before making changes.

Observe broker health read-only

Check whether brokers are up and connected to ZooKeeper (for ZooKeeper mode) or the KRaft metadata quorum. Use kafka-broker-api-versions.sh to query a broker without producing or consuming:

kafka-broker-api-versions.sh --bootstrap-server broker1:9092

A healthy broker returns a list of supported API versions. If the command times out, the broker is not listening. Check network reachability and broker logs under /var/log/kafka/server.log for FATAL or ERROR entries.

A read-only way to list topics and their metadata:

kafka-topics.sh --bootstrap-server broker1:9092 --list

Expected output (example):

__consumer_offsets
orders
payments

If the command hangs or returns Timed out waiting for a node assignment, the broker may be part of a failed cluster. Do not restart services blindly; first collect logs and timestamps.

Protect secrets and placeholders

Never paste real passwords or certificates into a terminal history or an article. Use environment variables:

export KAFKA_SASL_PASSWORD='<replace-with-password>'

In configuration snippets, use <your-bootstrap-server>, <your-topic>, <your-consumer-group>. When you must reference a file, use a path like /etc/kafka/secrets/client.truststore.jks rather than printing the content.

Safe Configuration Path

Configuration errors cause many Kafka failures. A safe path means: view the effective configuration read-only, change one setting at a time, and verify with a command that returns the new state.

Inspect broker configuration

Broker configuration lives in server.properties or environment variables. To see the current broker settings without restarting, use Kafka's dynamic config tools (Kafka 2.2+):

kafka-configs.sh --bootstrap-server broker1:9092 --entity-type brokers --entity-name 1 --describe

Example output:

Dynamic configs for broker 1 are:
  min.insync.replicas=2 sensitive=false synonyms={DYNAMIC_BROKER_CONFIG:min.insync.replicas=2}
  unclean.leader.election.enable=false sensitive=false synonyms={DYNAMIC_BROKER_CONFIG:unclean.leader.election.enable=false}

The output shows where each config comes from (DYNAMIC_BROKER_CONFIG, STATIC_BROKER_CONFIG, DEFAULT_CONFIG). Use this to determine if a change requires a rolling restart or can be applied dynamically.

Change one setting with verification

Example: increase replication.factor for a topic. First check the current topic config:

kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics --entity-name orders --describe

Example output:

Dynamic configs for topic orders are:
  retention.ms=604800000 sensitive=false synonyms={STATIC_TOPIC_CONFIG:retention.ms=604800000}
  min.insync.replicas=1 sensitive=false synonyms={DEFAULT_CONFIG:min.insync.replicas=1}

To change retention.ms to 3 days dynamically:

kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics --entity-name orders --alter --add-config retention.ms=259200000

Expected output:

Completed updating config for topic orders.

Verify:

kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics --entity-name orders --describe | grep retention.ms

If the change does not appear, verify the broker accepts dynamic configs and that you have the required ACLs.

Rollback safely

Always know how to revert. For a topic config, you can delete the dynamic override:

kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics --entity-name orders --alter --delete-config retention.ms

This restores the previous static or default value. Re-run the --describe command to confirm.

Verification and Diagnostics

After any change, verify the cluster, producers, and consumers behave as expected. Use read-only commands and examine metrics and logs.

Verify topic and partition metadata

Confirm the topic exists, has the right partitions and replicas, and leaders are available:

kafka-topics.sh --bootstrap-server broker1:9092 --describe --topic orders

Example output:

Topic: orders	PartitionCount: 3	ReplicationFactor: 3	Configs: segment.bytes=1073741824
	Topic: orders	Partition: 0	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3
	Topic: orders	Partition: 1	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1
	Topic: orders	Partition: 2	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2

Check that every partition has an in-sync replica (ISR) count equal to the replication factor. If ISR is less than replicas, a broker may be down or lagging.

Verify consumer group lag

Consumer group lag is a key health indicator. Use kafka-consumer-groups.sh:

kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group payments-group

Example output:

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID     HOST            CLIENT-ID
payments-group  payments        0          100             150             50   consumer-1-...  /10.0.0.5       consumer-1
payments-group  payments        1          200             250             50   consumer-1-...  /10.0.0.5       consumer-1

If LAG is consistently high, consumers are too slow or have stalled. Check for processing bottlenecks or rebalance loops.

Check broker logs for errors

Tail the broker log and filter for common error patterns:

tail -f /var/log/kafka/server.log | grep -E 'ERROR|FATAL|WARN'

Common error messages and what they mean:

• ERROR [ReplicaFetcherThread-0-1]: Error in fetch ... indicates broker-to-broker replication issues; check network and disk.
• WARN [RequestSendThread controllerId=...] Controller ... epoch ... may indicate controller failover; review controller logs.
• ERROR [Log partition=orders-0, dir=/var/lib/kafka/data] ... points to disk full or segment corruption; run df -h and kafka-log-dirs.sh --describe.

Use kafka-log-dirs.sh to inspect log directory health:

kafka-log-dirs.sh --bootstrap-server broker1:9092 --describe --broker-list 1

Example output:

{"broker":1,"logDirs":[{"logDir":"/var/lib/kafka/data","error":null,"partitions":[{"partition":"orders-0","size":123456,"offsetLag":0,"isFuture":false}]}]}

The error field should be null. If not, investigate the disk.

Failure Modes and Recovery

Kafka failures generally fall into a few categories: broker down, ZooKeeper/KRaft unavailable, disk full, producer timeouts, consumer rebalance issues, and configuration missteps. This section covers each with recovery steps.

Broker down

Symptom: producers fail with Connection to node -1 failed, consumers log Connection refused.

Diagnose: check broker process and port.

ps aux | grep kafka.Kafka
ss -tuln | grep 9092

If process is missing, check systemctl status kafka (if run as systemd service) or Docker/Kubernetes pod status. View the last lines of server.log:

tail -n 100 /var/log/kafka/server.log

Common causes:

• JVM out of memory: log contains java.lang.OutOfMemoryError. Increase heap in KAFKA_HEAP_OPTS and restart.
• Data directory corruption: log contains CorruptIndexException or FileNotFoundException. Restore from backup or delete the corrupted partition directory only after stopping the broker and ensuring replication elsewhere.
• Port conflict: another process on 9092. Change listeners or stop conflicting process.

Recovery: after fixing the root cause, restart the broker. Wait for it to join the cluster and catch up on replicas. Verify:

kafka-broker-api-versions.sh --bootstrap-server broker1:9092

and check ISR for topics. If a broker was down for a long time and fell out of ISR, its replicas need to catch up. Monitor kafka-topics.sh --describe until ISR returns to full.

ZooKeeper ensemble failure

Symptom: brokers cannot register or elect a controller; errors like ZooKeeper session expired or Unable to connect to zookeeper server within timeout.

Diagnose: check ZooKeeper status on each node:

echo stat | nc zookeeper1 2181 | grep Mode

Expected output:

Mode: follower

or Mode: leader.

If no output, ZooKeeper is not running. Start it with zkServer.sh start. If the ensemble is down entirely, you must restart at least a majority of nodes (2 out of 3 or 3 out of 5) to form a quorum. Check ZooKeeper logs under /var/log/zookeeper/zookeeper.out for exceptions.

After ZooKeeper recovers, Kafka brokers automatically reconnect. Verify broker registration:

kafka-broker-api-versions.sh --bootstrap-server broker1:9092

If brokers do not reconnect, restart them one by one.

KRaft metadata quorum failure

For KRaft clusters, if the controller quorum loses majority, no leader can be elected. Check kafka-metadata-quorum.sh:

kafka-metadata-quorum.sh --bootstrap-server broker1:9092 describe --status

Expected output:

ClusterId:          abc123...
LeaderId:           1
LeaderEpoch:        15
HighWatermark:      4000
MaxFollowerLag:     0
MaxFollowerLagTimeMs: 0
CurrentVoters:      [1,2,3]
CurrentObservers:   []

If LeaderId is -1, there is no active leader. Restore the quorum by ensuring at least a majority of controller nodes are running and can communicate. Check network ACLs and firewall rules between controllers.

Disk full

Symptom: brokers throw java.io.IOException: No space left on device or Error while writing to log. Producers receive RecordTooLargeException if disk blocks writes.

Diagnose:

df -h /var/lib/kafka/data

If usage is above 85%, take action. Identify large topics or partitions:

kafka-log-dirs.sh --bootstrap-server broker1:9092 --describe --broker-list 1

Recovery options:

• Reduce topic retention (but do it carefully):

  kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics --entity-name orders --alter --add-config retention.ms=86400000

This sets retention to 24 hours. Verify with --describe.

• Add more disk space or move log directories. To move a log directory, copy data to the new location while broker is stopped, update log.dirs, and restart.

• Delete unused topics after confirming with stakeholders. Use kafka-topics.sh --delete. Ensure delete.topic.enable=true on brokers.

Producer timeouts and errors

Common producer errors:

• org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for topic orders-0: 30000 ms has passed since batch creation
• Cause: broker unreachable or producer buffer full.
• Fix: check network, broker health, and increase delivery.timeout.ms if needed, but first check max.block.ms and batch.size.
• org.apache.kafka.common.errors.RecordTooLargeException
• Cause: record exceeds max.request.size on producer or message.max.bytes on broker.
• Fix: increase broker message.max.bytes and topic max.message.bytes and producer max.request.size to a consistent value. For example, set all to 10485760 (10 MB). Verify with kafka-configs.sh.
• org.apache.kafka.common.errors.NotLeaderForPartitionException
• Cause: producer is sending to a broker that is not the leader for the partition; usually transient during leader election.
• Fix: wait for leader election to complete; check kafka-topics.sh --describe to see current leader. Producers automatically retry.

Use kafka-producer-perf-test.sh to test producing:

kafka-producer-perf-test.sh --topic orders --num-records 1000 --record-size 100 --throughput 100 --producer-props bootstrap.servers=broker1:9092

Expected output includes records sent and latency percentiles. If records sent is 0, fix connectivity or configuration.

Consumer group rebalance loops

Symptom: consumers log (Re-)joining group frequently; lag grows; processing stops.

Cause: max.poll.interval.ms exceeded because processing takes too long; or session.timeout.ms too low; or too many partitions per consumer.

Diagnose: use consumer group describe:

kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group payments-group

If members keep changing, check consumer logs for rebalance triggers.

Fix:

• Increase max.poll.interval.ms to allow longer processing: set to 600000 (10 minutes).
• Adjust session.timeout.ms and heartbeat.interval.ms to suit network conditions.
• Reduce number of records per poll by lowering max.poll.records.
• Ensure each consumer handles partitions appropriately; use assign instead of subscribe if you need precise control.

After changing consumer config, restart consumers and watch lag over a few minutes. It should decline and stabilize.

Operations Checklist

Use this checklist to systematically handle Kafka errors. It follows the observe-change-verify-recover pattern.

1. Inventory the cluster

• Kafka version: kafka-topics.sh --version
• Broker status: kafka-broker-api-versions.sh --bootstrap-server <broker>:9092
• ZooKeeper or KRaft status: echo stat | nc <zookeeper-host> 2181 or kafka-metadata-quorum.sh --bootstrap-server <broker>:9092 describe --status
• Topic metadata: kafka-topics.sh --describe --topic <topic-name>
• Consumer group lag: kafka-consumer-groups.sh --describe --group <group-name>

Record all outputs and timestamps in a ticket or log.

2. Identify the failing component

• Producer: check producer logs for TimeoutException, RecordTooLargeException, NotLeaderForPartitionException
• Consumer: check consumer logs for rebalance, CommitFailedException, OffsetOutOfRangeException
• Broker: check server.log for FATAL, ERROR, disk errors
• ZooKeeper/KRaft: check for session expirations, leader elections, quorum loss

3. Make the smallest change

• Change one configuration value at a time
• Use dynamic config changes where possible to avoid restarts
• Document the before and after values
• Prepare a rollback command

4. Verify the result

• Re-run the diagnostic command that showed the problem
• Confirm expected output (e.g., lag decreasing, no error logs, leader available)
• If the problem persists, revert the change and investigate further

5. Recover and document

• If a change causes an incident, follow your rollback plan
• After resolution, update runbooks and monitoring alerts
• Add the error pattern to your knowledge base with the fix

Example runbook entry:

Error: Producer TimeoutException on topic orders
Cause: Broker 3 was down for maintenance, producer buffer filled up
Fix: Restart Broker 3, wait for ISR to recover, increase producer delivery.timeout.ms to 120000
Verification: kafka-producer-perf-test.sh sends 1000 records with 0 failures
Rollback: If new timeout causes higher latency, revert to 30000

Conclusion

Kafka common errors are manageable when you approach them with a disciplined workflow: inventory the version and environment, observe symptoms read-only, make the smallest safe change, verify the outcome, and document recovery. The examples in this guide provide concrete commands and expected outputs for the most frequent Kafka failures, from broker down to consumer rebalances.

Choose one low-risk verification from this article and practice it in a staging environment before you face a production incident. For instance, run kafka-consumer-groups.sh --describe --group <your-group> on a test cluster and confirm you can interpret the lag output. Record the current state before any change, and always keep a rollback path.

A reliable Kafka operation makes failures visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With these habits, Kafka common errors become routine checks rather than emergencies.