Kafka Upgrade and Migration with Practical Examples

Intro

Upgrading Apache Kafka is a routine task for platform teams, but a routine task can still take down a production pipeline if one step is missed. This guide is for developers, DevOps consultants, and technical startup teams who need to move Kafka from an older version to a newer one, or migrate a cluster to new infrastructure, without losing data or availability.

We focus on practical, example-driven procedures. Each section includes real commands, what output to expect, and what to do when the output does not match. You will learn how to inventory your current deployment, plan a safe upgrade path, verify the result, and recover if something goes wrong. We cover Kafka 2.x to 3.x upgrades and common migration scenarios, but the principles apply to other versions as well.

Operational safety is the core theme: observe before you change, keep changes small, use placeholders instead of real secrets, and always know how to get back to a working state.

Version and Environment Inventory

Before you touch anything, you need an accurate picture of what is running. This means recording the Kafka version, the deployment method, the broker configuration, and the client library versions that connect to your cluster. A mistake here can lead to incompatible upgrades or a failed migration.

Read-Only Observation

Start with read-only commands. Never run a change command until you have captured the current state and saved it somewhere safe. For a Kafka broker, the version is usually available from the kafka-broker-api-versions.sh script or by checking the running broker process.

Run this on a broker host:

./bin/kafka-broker-api-versions.sh --bootstrap-server broker1:9092 | grep -A 5 "broker1"

Expected output (example for Kafka 2.8.1):

broker1:9092 (id: 0 rack: null) -> (
  Produce(0): 0 to 8 [usable: 8],
  Fetch(1): 0 to 11 [usable: 11],
  ListOffsets(2): 0 to 5 [usable: 5],
  ...
)

The exact API versions are less important than the fact that the broker is reachable and responding. If you see a timeout or connection refused, stop and fix connectivity before proceeding.

To get the exact broker version from the host, check the file libs/kafka_2.13-<version>.jar or run:

ls /opt/kafka/libs/ | grep kafka_2.13

Example output:

kafka_2.13-2.8.1.jar

This tells you the Scala version (2.13) and Kafka version (2.8.1). Note that Kafka 3.x uses Scala 2.13 only; 2.x may use 2.12 or 2.13.

Topology and Environment Details

Record the following in a spreadsheet or runbook:

• Number of brokers, their hostnames, and broker IDs.
• Controller configuration: who is the active controller? (ZooKeeper for Kafka 2.x, KRaft for 3.x+).
• Client applications and their library versions (e.g., Java clients using kafka-clients 2.8.0, Python using confluent-kafka-python 1.8.2).
• Security configuration: SSL/TLS, SASL, ACLs.
• Topic replication factors and partition counts for critical topics.

Example: use kafka-topics.sh to list topics and describe one critical topic:

./bin/kafka-topics.sh --bootstrap-server broker1:9092 --list
./bin/kafka-topics.sh --bootstrap-server broker1:9092 --describe --topic orders

Expected output for --describe:

Topic: orders	PartitionCount: 6	ReplicationFactor: 3	Configs: min.insync.replicas=2
	Topic: orders	Partition: 0	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3

This shows the topic has 6 partitions, RF=3, min ISR=2, and all replicas are in sync. If any partition shows an ISR smaller than the replica set, resolve that before upgrading because a broker restart will temporarily remove more replicas.

Prerequisites for Upgrade

• Backup configuration: copy server.properties, log4j.properties, and any JAAS files to a safe location.
• Client compatibility: verify that your client libraries can talk to the target Kafka version. Kafka generally maintains backward compatibility: clients can be newer or older by one major version, but going from Kafka 0.10 clients to Kafka 3.x brokers will not work. Check the documentation.
• Disk space: ensure enough disk for rolling restarts and possible log growth.
• Maintenance window: coordinate with consumers and producers.

Smallest Justified Change

Do not upgrade all brokers at once. Use a rolling upgrade: upgrade one broker, wait, test, then proceed. The same applies to configuration changes: change one parameter at a time and verify.

Safe Configuration Path

The upgrade itself involves changing the broker configuration and replacing binaries. Follow a sequence that minimizes risk and allows rollback at every step.

Step-by-Step Upgrade Plan

Assume you are upgrading from Kafka 2.8.1 to Kafka 3.1.0 on three brokers. The official recommendation is to upgrade the brokers first, then the clients if needed, and then enable new features if desired.

1. Stop one broker gracefully.

Graceful shutdown ensures the broker leaves the ISR lists cleanly.

./bin/kafka-server-stop.sh

Wait until the process ends and then verify it is not in the cluster by checking the controller log or running:

./bin/kafka-broker-api-versions.sh --bootstrap-server broker1:9092 | grep broker1

(Should return nothing or connection refused.)

2. Replace the Kafka distribution on that broker.

Keep the old installation in a separate directory, e.g., /opt/kafka-old. Install the new version in /opt/kafka. Copy your saved server.properties and other configuration files into the new config directory. Do not copy log directories.

3. Start the upgraded broker.

./bin/kafka-server-start.sh -daemon config/server.properties

Check the log file logs/server.log for startup success. Look for a line like:

[2025-03-01 14:30:00,123] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)

4. Verify the broker rejoined the cluster.

Run the API versions command again:

./bin/kafka-broker-api-versions.sh --bootstrap-server broker1:9092 | grep broker1

You should now see version details. Also check the topic under-replication status:

./bin/kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions

If this returns nothing, all replicas are in sync.

5. Repeat for the other brokers one at a time.

Do not proceed to the next broker until the current one is healthy and all replicas are caught up.

Configuration Changes to Avoid During Upgrade

Do not change broker configuration settings at the same time as the version upgrade. For example, do not change num.partitions or default.replication.factor during the rolling upgrade. If you need to make a configuration change, do it after all brokers are upgraded and stable, or beforehand, but not simultaneously.

Blast Radius and Recovery

If a broker fails to start after upgrade, you can roll back by stopping it, restoring the old Kafka directory (e.g., rename /opt/kafka to /opt/kafka-bad, and /opt/kafka-old to /opt/kafka), and restarting with the old version. The broker will rejoin the cluster and replicate any missed data from the other brokers, as long as the log directory is intact. Do not delete the log directory unless you are prepared to lose data or rebuild from replicas.

Verification and Diagnostics

After all brokers are upgraded, you must verify that the cluster is healthy and that clients can produce and consume.

Cluster Health Checks

1. Check broker count and versions.

./bin/zookeeper-shell.sh zookeeper1:2181 <<< "ls /brokers/ids"

(For KRaft mode, use kafka-metadata-shell.sh.)

Expected output:

[0, 1, 2]

All broker IDs should be present.

2. Check controller status.

In the broker logs or via JMX, find which broker is the controller. Ensure there is exactly one active controller. If you see controller failover loops, investigate network or ZooKeeper issues.

3. Check under-replicated and offline partitions.

./bin/kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions
./bin/kafka-topics.sh --bootstrap-server broker1:9092 --describe --unavailable-partitions

Both should return no output. If any partitions are under-replicated, wait or investigate the broker logs.

Client Smoke Test

Use the console producer and consumer to test basic functionality:

Produce:

echo "test message $(date)" | ./bin/kafka-console-producer.sh --bootstrap-server broker1:9092 --topic test-topic --producer-property acks=all

Consume:

./bin/kafka-console-consumer.sh --bootstrap-server broker1:9092 --topic test-topic --from-beginning --max-messages 1

Expected output: the same message string you sent.

Monitoring Metrics

Key metrics to watch during and after upgrade:

• kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions should be 0.
• kafka.controller:type=KafkaController,name=ActiveControllerCount should be 1.
• kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce should not drop significantly compared to baseline.
• Consumer lag for critical consumer groups should not increase beyond normal.

If any metric is out of range, diagnose before opening the cluster to full traffic.

Failure Modes and Recovery

Even with careful planning, things can go wrong. Here are common failure modes and how to recover.

Broker Fails to Start After Upgrade

Symptoms: broker process exits immediately, server.log shows ClassNotFoundException, NoSuchMethodError, or incompatible configuration.

Why it happens:

• Mixed Scala versions: using a 2.12-compiled jar with 2.13 runtime or vice versa.
• Configuration key renamed or removed in the new version.
• Missing dependent jar files if you copied the installation incorrectly.

Recovery:

1. Check the error in logs/server.log.
2. If it is a configuration issue, fix the server.properties file (e.g., remove deprecated zookeeper.connect if using KRaft).
3. If it is a binary mismatch, ensure you are using the correct Kafka distribution for your Scala version.
4. If all else fails, roll back to the old version by stopping the broker, restoring the old directory, and restarting. The broker will catch up with replicas.

Under-Replicated Partitions Persist

Symptoms: kafka-topics.sh --describe --under-replicated-partitions lists partitions for a long time after upgrade.

Why it happens:

• Broker was down too long and fell out of ISR.
• Network partition or disk I/O issues.
• Replica reassignment in progress.

Recovery:

• Check broker logs for replication errors.
• Check disk usage: if a broker's disk is full, replication will stall. Free up space.
• Ensure all brokers are running and can communicate on the replication port (default 9092 for inter-broker, or 9093 if different).
• If needed, force reassignment or restart the broker to trigger replication.

Client Compatibility Issues

Symptoms: Clients throw UnsupportedVersionException or similar after upgrade.

Why it happens:

• Clients are too old relative to the broker.
• Clients are using a removed API (e.g., ZooKeeper-based consumer groups).

Recovery:

• Upgrade the client library to a version compatible with the broker. Check the Kafka compatibility matrix.
• For old consumer groups using ZooKeeper, migrate to the new consumer group protocol before upgrading brokers beyond 3.x, or keep a compatibility broker.

Data Loss or Corruption

Symptoms: Some messages missing after upgrade, or broker unable to load log segments.

Why it happens:

• Disk issues during upgrade.
• Unclean leader election enabled (unclean.leader.election.enable=true) combined with broker failures.
• Manually deleting log directories.

Recovery:

• If log segments are corrupt, you may need to restore from backup or use kafka-reassign-partitions.sh to move affected partitions to a healthy broker.
• To avoid this, always set unclean.leader.election.enable=false (default) and ensure min.insync.replicas is set appropriately for critical topics.
• Always keep a backup of log directories before destructive actions.

Migration Scenarios

Upgrading in place is one thing; migrating to new hardware or a new cluster is another. Here we cover two common migration approaches.

MirrorMaker 2 for Cross-Cluster Replication

If you are moving to a new cluster (e.g., new data center or managed Kafka service), MirrorMaker 2 can replicate topics and consumer offsets.

Example setup:

Define source and target clusters in mm2.properties:

clusters = source, target
source.bootstrap.servers = source-broker1:9092
target.bootstrap.servers = target-broker1:9092
source->target.enabled = true
source->target.topics = orders, payments
target->source.enabled = false

Run MirrorMaker 2:

./bin/connect-mirror-maker.sh mm2.properties

Monitor replication using consumer group offsets or the MirrorMaker 2 metrics. After the data is replicated and consumers are switched to the target cluster, you can decommission the source.

Cluster Expansion and Reassignment

If you are adding brokers to an existing cluster and want to move topics to them, use kafka-reassign-partitions.sh.

1. Generate a reassignment plan.

Create a JSON file with the topics to move:

{"topics": [{"topic": "orders"}], "version":1}

./bin/kafka-reassign-partitions.sh --bootstrap-server broker1:9092 --topics-to-move-json-file topics.json --broker-list "3,4,5" --generate

2. Review and execute the plan.

Copy the generated plan into reassign.json and execute:

./bin/kafka-reassign-partitions.sh --bootstrap-server broker1:9092 --reassignment-json-file reassign.json --execute

3. Verify completion.

./bin/kafka-reassign-partitions.sh --bootstrap-server broker1:9092 --reassignment-json-file reassign.json --verify

Expected output: Reassignment of partition orders-0 completed successfully etc.

Common Pitfalls and How to Avoid Them

Many upgrade and migration issues are caused by avoidable mistakes. Here are the most common pitfalls, why they happen, and how to prevent or recover from them.

1. Not Checking Client Compatibility Before Upgrading Brokers

Why it happens: Teams focus on the broker version and forget that producers and consumers have their own version dependencies.

How to avoid: Before upgrading brokers, list all client applications and their library versions. Check the Kafka compatibility matrix to ensure they can talk to the new broker version. If not, plan to upgrade clients first or simultaneously.

Recovery: If clients break after broker upgrade, you may need to roll back the brokers until clients are upgraded, or upgrade clients quickly.

2. Changing Multiple Variables at Once

Why it happens: To save time, teams upgrade the version and also tweak configuration settings or hardware at the same time.

How to avoid: Strictly separate version upgrades from configuration changes. Do one type of change, fully test, then do the other.

Recovery: If a combined change causes problems, you will have a harder time identifying the cause. Roll back one variable at a time to isolate.

3. Ignoring the Controller and Metadata Quorum

Why it happens: In Kafka 2.x with ZooKeeper, teams may forget to check ZooKeeper health. In Kafka 3.x with KRaft, they may not understand the new controller quorum.

How to avoid: Always verify controller and ZooKeeper (or KRaft) health before and after upgrade. If using ZooKeeper, ensure the ZooKeeper ensemble is healthy and has a leader. If using KRaft, ensure a quorum of controllers is running.

Recovery: If ZooKeeper is unhealthy, fix it before touching brokers. If KRaft loses quorum, restore the missing controllers.

4. Not Having a Tested Rollback Plan

Why it happens: Teams assume the upgrade will go smoothly and do not prepare a rollback procedure.

How to avoid: Write a rollback plan before starting. Include commands to stop the new version, restore the old binaries and configuration, and restart. Test the rollback in a staging environment if possible.

Recovery: If you did not prepare and something fails, your fallback is to restore from backups or rebuild from replicas, which may be slower.

5. Forgetting to Monitor Consumer Lag

Why it happens: After a broker upgrade, consumer groups may rebalance, causing temporary lag. Teams forget to watch lag and assume the system is fine.

How to avoid: Set up alerts on consumer lag for critical groups. After upgrade, watch lag for at least 24 hours to catch delayed issues.

Recovery: If lag is high, check consumer rebalance logs, broker I/O, and network. You may need to restart consumers or scale up.

Operations Checklist

Use this checklist for every Kafka upgrade or migration. It is designed for a single accountable owner to drive each step and verify completion.

Each owner is a single accountable individual, not a group. The revisit frequency indicates when they should re-evaluate the outcome if needed.

Conclusion

Upgrading or migrating Apache Kafka does not have to be a high-risk event. The key is to approach it methodically: inventory your environment, plan a safe rolling upgrade, verify each step, and have a tested rollback path. The examples in this guide provide a starting point, but always adapt commands and checks to your specific Kafka version and deployment.

Most failures come from skipping the basics: not checking client compatibility, changing too many things at once, or lacking a rollback plan. By following the practices outlined here, you can minimize downtime, avoid data loss, and keep your streaming platform healthy.

Next, pick a low-risk verification task from the checklist, such as running the read-only version command and recording the output. Then, schedule a maintenance window for a rolling upgrade of one broker in a staging cluster, practice the rollback, and refine your runbook. As you gain confidence, expand to production.

A reliable upgrade workflow makes failures visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.