Kafka Performance Tuning with Practical Examples: A Hands-On Guide for Operators

Intro

Kafka performance problems rarely announce themselves. They show up as p99 latency climbing after a deploy, consumers falling behind during peak traffic, or brokers pausing for GC at the worst moment. Tuning Kafka without a baseline is guesswork; tuning with a baseline but no recovery plan is gambling. This guide walks through a structured approach: observe first, change one thing, verify the result, and know how to roll back.

We will assume Apache Kafka 3.x on Linux, managed with the standard shell tools (kafka-configs.sh, kafka-producer-perf-test.sh, and friends). The same logic applies to Confluent Platform, Strimzi, and MSK, but command paths and defaults may differ. Every example uses placeholders like $BROKER and $TOPIC to avoid pasting real cluster details. Do not reuse commands blindly, especially those that alter broker configuration or topic retention.

The workflow in this article maps to the lifecycle of a performance issue: inventory the environment, change configuration safely, verify with metrics, diagnose failures, and follow an operational checklist. Along the way we will hit the most common traps: misreading acks, ignoring consumer lag, over-provisioning partitions, and tuning from folklore instead of measurements.

Version and Environment Inventory

Before touching any setting, know what you are running. Different Kafka versions change defaults, deprecate options, and fix performance bugs. For example, unclean.leader.election.enable defaulted to true before 2.4 and false after; running an old client against a new broker can silently use old defaults. Capture the following in a runbook, not your head:

# Broker version
kafka-broker-api-versions.sh --bootstrap-server $BROKER | head -n 1

# Topic configuration
kafka-configs.sh --bootstrap-server $BROKER --entity-type topics --entity-name $TOPIC --describe

# Broker configuration (single broker)
kafka-configs.sh --bootstrap-server $BROKER --entity-type brokers --entity-name 0 --describe

Expected output for the first command looks like $BROKER:9092 (id: 0 rack: null) -> (...). If the command hangs or returns a TimeoutException, the broker is not reachable from your host, which is a prerequisite for every later step.

Record the deployment topology: number of brokers, number of partitions for critical topics, replication factor, and whether you use rack awareness. A cluster with three brokers and a topic of 30 partitions has a very different tuning profile than a single-broker development box. Write down the current values alongside a timestamp. Later, when kafka-topics.sh --describe shows an extra partition or a different min.insync.replicas, you will know what changed.

Privilege boundaries matter too. Only use credentials that can read configs for observation. Configuration changes need broker admin rights, and if you are on MSK, the AWS IAM policy must allow kafka:AlterConfig on the cluster. Keep separate read-only and write-enabled profiles so an accidental --alter does not go through with production privileges.

Safe Configuration Path

The safest performance change is the one you can reverse in under a minute. Dynamic broker configs via kafka-configs.sh take effect without a restart and can be reverted just as quickly. Static configs in server.properties require a rolling restart, which is a heavier operation. Prefer dynamic changes when the option supports them.

A concrete example: producer throughput is limited by unclean leader election during broker restarts. Instead of flipping unclean.leader.election.enable globally, check the current setting and test in a staging topic.

Observe first:

kafka-configs.sh --bootstrap-server $BROKER --entity-type topics --entity-name $TOPIC --describe | grep -i unclean

Expected output: unclean.leader.election.enable=true sensitive=false synonyms={...}. If the output is empty, the topic is using the broker default (false in Kafka 2.4+).

Change one scoped item at a time. To allow unclean leader election for only one non-critical topic:

kafka-configs.sh --bootstrap-server $BROKER --alter --entity-type topics --entity-name $TOPIC \
  --add-config unclean.leader.election.enable=true

Verification:

kafka-configs.sh --bootstrap-server $BROKER --entity-type topics --entity-name $TOPIC --describe

Look for unclean.leader.election.enable=true in the output. Rollback is the inverse:

kafka-configs.sh --bootstrap-server $BROKER --alter --entity-type topics --entity-name $TOPIC \
  --delete-config unclean.leader.election.enable

Deleting the override returns the topic to the broker default. Blast radius: any producer or consumer using this topic can now see data loss during a leader failover if replicas are out of sync. Only enable this on topics where data loss is acceptable, like a non-critical telemetry stream.

For static changes, such as increasing num.network.threads in server.properties, plan for a rolling restart. Check the current value with grep num.network.threads $KAFKA_HOME/config/server.properties. If you double it, watch NetworkProcessorAvgIdlePercent before and after. If idle percent stays near 1.0, the network threads were not the bottleneck, and you should revert to avoid wasting memory.

Verification and Diagnostics

Verification without a baseline is noise. Start with the built-in performance test tools before making changes, then repeat after. Run a producer throughput test on a dedicated topic with a clean consumer group.

kafka-producer-perf-test.sh \
  --topic perf-test \
  --num-records 1000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=$BROKER acks=1 linger.ms=10 batch.size=65536

Expected output includes records/sec, MB/sec, and average latency. Note the numbers. Now change one broker config, rerun the same test, and compare. If throughput drops by more than 10 percent or p99 latency triples, the change was harmful.

For diagnosing a live cluster, kafka-consumer-groups.sh is your first stop. Consumer lag is the single most useful metric for end-to-end performance.

kafka-consumer-groups.sh --bootstrap-server $BROKER --describe --group $GROUP

Look for the LAG column. A steadily increasing lag means consumers cannot keep up. Check consumer-side configs like fetch.min.bytes, fetch.max.wait.ms, and max.poll.records. If consumers are pausing for long garbage collection, increasing fetch.max.wait.ms may help batch more data per request.

Broker-side, kafka-broker-api-versions.sh shows active connections and request rates per API. For deep metrics, use JMX with Prometheus and Grafana. Critical broker metrics include:

• kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce (p99)
• kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
• kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec
• kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs
• java.lang:type=GarbageCollector,name=G1 Old Generation,attribute=CollectionTime

Set up alerts on producer request p99 above 500 ms, consumer lag over 10000 for critical groups, and GC pause time over 100 ms. Without these, you are flying blind.

Failure Modes and Recovery

Kafka can fail in ways that look like performance problems. Here are three common ones and how to recover.

Producer timeouts due to min.insync.replicas

If producers get NotEnoughReplicasException, the cluster is rejecting writes because fewer than min.insync.replicas brokers are alive or in sync. Check:

kafka-topics.sh --bootstrap-server $BROKER --describe --topic $TOPIC

If the ISR column shows fewer than min.insync.replicas, either a broker is down or a replica is lagging. Recovery: bring the broker back online, or lower min.insync.replicas temporarily on that topic (knowing you accept lower durability). The change is dynamic and reversible.

Consumer group rebalance storms

If consumers repeatedly join and leave the group, look for max.poll.interval.ms exceeded. A slow processing task causes the broker to kick the consumer. Logs will show MemberHeartbeatException or RebalanceInProgressException. Increase max.poll.interval.ms and session.timeout.ms for that consumer group, but first check if the processing is simply too slow for Kafka's default. Sometimes the fix is to process less per poll, not to hide the rebalance.

Disk full on a broker

A full log directory causes producers to fail with KafkaStorageException. Check disk usage with df -h. Recovery depends on the topic's retention policy. You can reduce retention dynamically:

kafka-configs.sh --bootstrap-server $BROKER --alter --entity-type topics --entity-name $TOPIC \
  --add-config retention.ms=3600000

This keeps only the last hour of data. Then scale up disk or add more brokers. After the immediate pressure is off, revisit the retention policy for permanent values.

For each failure, the recovery path should be tested in staging before it is needed in production. A recovery plan that has never been run is a hypothesis.

Operations Checklist

Use this checklist before and after any Kafka tuning change. Assign an owner and a review cadence for each item.

Each item has a single accountable owner, not a committee. The frequency is explicit: daily disk checks, weekly lag review, quarterly rollback drills. If an item fails, the owner is the first to know.

Common Pitfalls

Kafka tuning folklore is full of traps. Here are the ones that waste the most time.

Pitfall 1: Tuning num.partitions too high

Why it happens: More partitions means more parallelism, right? But each partition is a unit of replication, and too many partitions increase memory use, file handles, and end-to-end latency. A cluster with 100,000 partitions can become unusable.

How to avoid: Calculate expected throughput per partition (usually 10-50 MB/s) and size partitions to total throughput. Monitor UnderReplicatedPartitions and broker memory if you increase partitions.

Pitfall 2: Using acks=all without understanding the cost

Why it happens: acks=all gives the strongest durability guarantee, so people set it everywhere. But it doubles write latency compared to acks=1 in a three-broker cluster. If your use case can tolerate losing the last few messages on a rare broker crash, acks=1 may be appropriate.

How to avoid: Measure with kafka-producer-perf-test.sh under both settings. Document the trade-off in the topic's runbook.

Pitfall 3: Ignoring consumer fetch configuration

The default fetch.min.bytes=1 means consumers get data as soon as a single byte is available, which can cause many small requests and high CPU. Increasing fetch.min.bytes to 1024 or higher adds a small amount of latency but improves throughput and reduces broker load.

How to avoid: Benchmark consumer throughput with different fetch.min.bytes values. Use kafka-consumer-perf-test.sh if available, or a custom consumer that measures records/sec.

Pitfall 4: Changing broker configs without restarting or vice versa

Some configs are read only at broker startup. Changing num.io.threads in server.properties and running kafka-configs.sh --alter does nothing, because num.io.threads is read-only at runtime. Conversely, changing unclean.leader.election.enable dynamically but expecting it to apply to all existing topics without a per-topic override is wrong.

How to avoid: Check the Kafka documentation for each config's Update Mode. Dynamic configs say per-broker or cluster-wide; static configs say read-only. Use kafka-configs.sh --describe to see what actually took effect.

Pitfall 5: Treating consumer lag as a problem to hide

If lag is high, some people increase max.poll.records or decrease max.poll.interval.ms to make the consumer catch up faster. But if the consumer's processing is the bottleneck, this just causes rebalances or out-of-memory errors. The root cause may be a slow downstream database or a CPU-bound transformation.

How to avoid: Profile the consumer application. Use jstack or a profiler to see where time is spent. If it is in user code, Kafka tuning will not fix it.

Conclusion

Kafka performance tuning is a loop: measure, change, verify, and prepare to roll back. The commands in this article give you the raw material, but the discipline is what prevents outages. Version-scoped inventory, dynamic config changes where possible, metric thresholds with owners, and tested recovery paths separate a calm tuning session from a firefight.

Next step: choose one low-risk topic in your staging cluster, capture the current configs and baseline throughput, make a single dynamic change (such as unclean.leader.election.enable), rerun the performance test, and compare. If the numbers move in the right direction, roll the change out with a monitoring alert tied to the owner. If not, delete the override and document why.

Do not forget the dependencies: Kafka rarely operates alone. If you tune Kafka but write from NiFi, read through Spark, or archive to HDFS, measure the end-to-end path, not just broker throughput. A Kafka cluster that is fast in isolation but slow in the hands of its consumers is not fast at all.