Kafka troubleshooting with practical examples: practical implementation guide

A hands-on guide to diagnose and fix common Kafka problems. Learn how to read logs, run safe commands, confirm root causes, and execute step-by-step recovery workflows.

Intro

Kafka issues often surface first as timeouts, growing consumer lag, or scary log lines during peak traffic. This guide gives you a practical, repeatable way to diagnose and fix the most common failures without risking data loss. You will learn how to read broker and client logs, run safe commands, confirm root causes, and execute step-by-step recovery workflows. The examples are designed for developers, DevOps consultants, and technical startup teams running Kafka alongside tools like NiFi, Apache Spark, HDFS, and Apache Airflow.

Workflow Overview

Use this triage-to-recovery path to stay methodical under pressure:

1. Stabilize and scope

• Freeze risky changes. Pause noisy automation if it is masking symptoms.
• Scope blast radius: single client, topic, broker, AZ, or cluster-wide.

1. Capture signals

• Save the last 15 to 60 minutes of logs from clients and brokers.
• Snapshot key metrics: request latencies, consumer lag, under-replicated partitions, broker disk, file descriptors, CPU, and network.

1. Classify the symptom

• Producer timeouts or high latency
• Consumer lag growth or stuck partitions
• Under-replicated partitions or ISR shrink
• Authentication or authorization failures
• Broker start failures or controller flaps

1. Run read-only checks first

• Describe topics, groups, brokers. Inspect configs and partition states.
• Confirm network reachability and DNS.

1. Form a root-cause hypothesis

• Client-side vs broker-side vs infra (network, disk) vs security.

1. Apply the smallest safe change

• Prefer reversible changes: throttles, pausing consumers, rolling restarts.
• Avoid deletes, force leadership, or config changes that reduce safety unless it is a last resort.

1. Verify and unwind

• Re-check metrics and logs. Remove temporary throttles and toggles.

1. Document findings

• Capture the minimal set of facts that explain the failure and fix.

Common Symptoms and Logs

Match these log lines and metrics to likely causes:

Producer-side symptoms

Meaning: Network congestion, overloaded broker, incorrect bootstrap, or bad DNS.

Meaning: ISR too small for acks=all, replicas offline or slow disk.

Meaning: Message exceeds topic or broker max message size.

• Error: Request timed out, TimeoutException
• Error: NotEnoughReplicas or NotEnoughReplicasAfterAppend
• Error: RecordTooLargeException

Consumer-side symptoms

Meaning: Processing too slow, rebalance loops, or partition stuck on a dead broker.

Meaning: Offsets expired due to retention or compacted away.

Meaning: Group churn, max.poll.interval.ms exceeded, or unstable broker.

• Lag increases even though consumer is running
• Error: OffsetOutOfRange
• Rebalance storms in logs

Broker and cluster symptoms

Meaning: One or more replicas are lagging or offline. Check disk, network, CPU.

Meaning: Metadata instability, network partitions, or quorum issues.

Meaning: Broker saturation. Look at CPU and I/O.

Meaning: Followers cannot keep up due to slow disk or network.

• Under-replicated partitions > 0
• Controller elections or flapping controller logs
• SocketServer, RequestHandlerPool backlog
• ReplicaFetcherThread warnings

Security symptoms

Meaning: Mismatch in mechanism or cert trust issues.

Meaning: ACLs are missing or principal is wrong.

• Client authentication failed, SASL handshake failed, SSL handshake error
• Authorization failed for Create/Write/Read

Storage and OS symptoms

Meaning: Retention too long, compaction backlog, or log cleanup stuck.

Meaning: File descriptor limit too low for topic-partition count.

• Disk at or near 100 percent
• Too many open files

Root-Cause Checks

Client checks

• Verify bootstrap servers, security.protocol, and auth settings.
• Measure producer retry rate, batch sizes, and compression effectiveness.
• For consumers, confirm max.poll.interval.ms, max.poll.records, and processing time.

Network checks

• Test DNS resolution of brokers and advertised.listeners.
• Check cross-AZ or cross-DC latency and packet loss.
• Confirm firewall or security group rules on broker ports.

Broker checks

• Describe brokers and topics to find leaders, ISR, and replica states.
• Look for thread pool starvation, GC pauses, and page cache pressure.
• Validate min.insync.replicas vs acks=all behavior.

Storage checks

• Ensure adequate free space on log.dirs. Kafka needs headroom for segment rollover and compaction.
• Monitor disk throughput and iowait. Slow disks cause ISR shrink.

Metadata and quorum checks

• Verify controller stability. If using ZooKeeper, check session stability. If using KRaft, check quorum voters and election logs.
• Watch for frequent leader changes which can cascade into client timeouts.

Security checks

• Confirm principals, mechanisms, and truststores. Align client and broker configs.
• Check ACLs for required operations on the target topics and groups.

Safe Commands and Recovery

Use read-only commands first. Prefer --describe and --list before making changes. Replace <...> placeholders with your values.

Topic and broker state

• List topics:

kafka-topics.sh --bootstrap-server <broker:9092> --list

• Describe a topic in detail:

kafka-topics.sh --bootstrap-server <broker:9092> --describe --topic <topic>

• Describe brokers (via metadata dump if available) or check server logs for node IDs and listener info.

Consumer groups

• List groups:

kafka-consumer-groups.sh --bootstrap-server <broker:9092> --list

• Describe lag for a group:

kafka-consumer-groups.sh --bootstrap-server <broker:9092> --describe --group <group>

• Safely reset offsets (dry run first):

kafka-consumer-groups.sh --bootstrap-server <broker:9092> \
  --group <group> --topic <topic> --reset-offsets --to-latest --dry-run

• Apply only after verifying:

kafka-consumer-groups.sh --bootstrap-server <broker:9092> \
  --group <group> --topic <topic> --reset-offsets --to-latest --execute

Configs

• Inspect topic configs:

kafka-configs.sh --bootstrap-server <broker:9092> --entity-type topics \
  --entity-name <topic> --describe

• Update a topic config safely (example: temporary retention increase):

kafka-configs.sh --bootstrap-server <broker:9092> --entity-type topics \
  --entity-name <topic> --alter --add-config retention.ms=<value>

Partition movement

• Generate and apply a reassignment plan with throttle. Always review the plan and apply during a low-traffic window.

Throttling

• Temporarily throttle producers or use quotas to reduce load while recovering.

Caution

• Avoid force leader elections and disabling safety configs unless you accept data loss risk. Document risks before proceeding.
• Prefer rolling any broker change. Change one broker, validate, then continue.

Practical Examples

Example 1: Consumer lag spike after a deploy Goal: Restore steady consumption without losing messages.

1. Verify lag and assignment

• Command:

kafka-consumer-groups.sh --bootstrap-server <broker:9092> --describe --group <group>

• Look for partitions with high lag and check if the consumer is assigned.

1. Check consumer logs

• Signs: Rebalance loops, CommitFailedException, processing taking too long.

1. Hypothesize

• The deploy increased processing time or reduced poll frequency, hitting max.poll.interval.ms.

1. Safe actions

• Increase max.poll.interval.ms moderately or reduce per-record work.
• Temporarily scale out consumers to match partition count.
• If lag is ancient and acceptable to skip, plan an offsets reset with --dry-run first.

1. Verify

• Lag should trend down. Watch rebalance frequency and consumer processing time metrics.

Example 2: Under-replicated partitions after a broker disk fills Goal: Restore ISR and replication health safely.

1. Confirm scope

• Command:

kafka-topics.sh --bootstrap-server <broker:9092> --describe --topic <topic>

• Monitor cluster metric: under-replicated partitions.

1. Broker inspection

• Check disk free space on the affected broker. Review server logs for disk errors.

1. Stabilize

• Temporarily throttle producers or pause high-volume ingestion.
• Consider temporarily increasing topic retention.ms only if you need headroom to compact or move data later.

1. Remediate disk

• Free space safely or add capacity on log.dirs.
• Avoid manual deletion of Kafka log segments unless you fully understand the implications.

1. Recover and verify

• Restart the affected broker if required. Confirm ISR grows back.
• Watch ReplicaFetcherThread and network throughput.

Example 3: Producer timeouts and RecordTooLargeException Goal: Restore producer throughput without breaking consumers.

1. Confirm error

• Producer logs show RecordTooLargeException or similar.

1. Options

• Reduce message size by splitting payloads or increasing compression.
• If needed, adjust broker or topic limits cautiously:
• topic config: max.message.bytes
• broker configs that interact: message.max.bytes, replica.fetch.max.bytes
• Make changes incrementally and prefer a topic-level change over cluster-wide.

1. Verify end-to-end

• Ensure consumers can fetch larger messages. Validate no spike in fetch timeouts.

Example 4: Authentication failures after certificate rotation Goal: Restore client-broker trust.

1. Check logs

• Look for SSL handshake errors or SASL mechanism mismatches.

1. Validate configs

• Align security.protocol and mechanism on both sides.
• Verify keystore and truststore paths, passwords, and that the CA chain is present.

1. Test connectivity

• From the client host, confirm TCP reachability to the broker listener.

1. Verify fix

• Successful connection and stable produce-consume loop.

Example 5: Controller flapping and frequent leader changes Goal: Stabilize metadata to reduce client timeouts.

1. Observe

• Broker logs show frequent controller elections. Clients see NotLeaderForPartition and timeouts.

1. Check quorum stability

• For ZooKeeper: verify session stability and network between brokers and ZooKeeper.
• For KRaft: confirm voter set, connectivity, and that the controller node is healthy.

1. Remediate

• Fix network partitions, remove noisy restarts, and ensure clocks are in sync.
• Apply rolling restarts only after confirming quorum health.

1. Verify

• Controller becomes stable. Leader changes drop to normal levels.

Local Pilot Plan

Objective: Prove a small set of troubleshooting workflows locally before touching shared environments.

Scope

• Pick 2 scenarios: consumer lag spike and RecordTooLargeException.
• Use a single test topic with 3 partitions and a small replication factor.

Plan

1. Prepare tooling

• Scripts to capture logs and run describe commands quickly.
• A dashboard or simple charts for lag, latency, and under-replicated partitions.

1. Run scenarios

• Generate small messages, then inject a large message to trigger RecordTooLargeException.
• Slow down consumer processing to trigger lag growth.

1. Practice recovery

• For large records: split payloads or adjust topic max.message.bytes in a controlled way.
• For lag: tune consumer poll settings and scale out workers.

1. Measure

• Time to detect symptom: target under 2 minutes.
• Time to confirm root cause: target under 10 minutes.
• Time to safe recovery: target under 20 minutes.

1. Codify

• Turn the commands and checks into a short runbook with copy-paste snippets.

This narrow, measurable pilot is easy to inspect locally and builds confidence before cluster-wide changes.

Conclusion

A steady troubleshooting workflow turns Kafka incidents into predictable operations work. Start with read-only inspection, classify the symptom, test the smallest safe change, and verify. Practice with a local pilot so the team can execute under stress. Keep a short runbook of commands and log patterns. Over time, you will reduce recovery time and avoid risky last-resort actions.

Quick checklist

• Can I scope the blast radius fast?
• Do I have the last hour of client and broker logs?
• What error category am I in: producer, consumer, broker, security, or storage?
• Which read-only command will confirm my hypothesis?
• What is the smallest safe change, and how will I roll it back?
• Did metrics recover, and did I remove any temporary throttles?