Apache Kafka Backup & Restore: Production Strategies, Commands, and Validation

Operational Kafka backup requires matching a recovery strategy to explicit RPO and RTO targets, cluster topology (KRaft vs. ZooKeeper), and security posture. This guide covers five production methods—MirrorMaker 2, Kafka Connect Replicator, volume snapshots, tiered storage export, and kafka-dump-log—with version-scoped commands, verification signals, failure-mode diagnostics, and rollback procedures. All examples use $UPPER_SNAKE_CASE placeholders; replace them with your values before execution. Test every procedure in a non-production environment first; managed services (Confluent Cloud, Amazon MSK, Aiven, Redpanda) may restrict broker-level access or provide proprietary tooling.

⚠️ Blast Radius: Never run backup or restore commands against a production cluster without a validated runbook, isolated staging test, and on-call escalation path.

Prerequisites & Assumptions

Kafka Versions Covered

• 3.5.x: KRaft production-ready (KIP-590), no tiered storage GA.
• 3.6.x: Tiered storage GA (KIP-405), improved metadata quorum snapshots (KIP-848).
• 3.7.x LTS: Current long-term support; includes all 3.6 features plus stability fixes.

Topology Distinctions

• KRaft: Controller quorum stores metadata in the __cluster_metadata partition 0; backup requires a quorum snapshot or controller log capture.
• ZooKeeper: Ensemble stores cluster metadata under /cluster/meta, /brokers, /config; backup via zkCli.sh export.

Infrastructure Requirements

• Block storage: AWS EBS, GCP Persistent Disk, Azure Managed Disk, or Kubernetes PVC (Portworx, Longhorn, Rook-Ceph).
• Object storage: S3, GCS, or Azure Blob for tiered storage export and snapshot archival.
• Network: Dedicated VPC peering or Transit Gateway for cross-region replication; bandwidth ≥ peak produce throughput (measure via kafka.server.BrokerTopicMetrics.MessagesInPerSec).

Access & Security

• Broker CLI access via kafka-*.sh scripts on an admin host.
• Admin API tokens or SASL/SSL client properties ($CLIENT_PROPS) with DescribeConfigs, Read, ClusterAction ACLs for the backup principal.
• IAM roles for cloud snapshot APIs (ec2:CreateSnapshot, s3:PutObject).
• Schema Registry read/write for subject export/import.
• Encryption at rest (EBS encryption, PVC storage class) and in transit (TLS 1.2+).

Capacity Planning

• Local staging: 1.5×–2× total log.dir size for segment exports and snapshot staging.
• Retention: Minimum 2× RPO window for incremental methods; 7–30 days for snapshot archives.

Architecture & Strategy Selection

Choose a method by mapping your RPO/RTO to the decision matrix below. All RPO/RTO values assume healthy clusters and provisioned bandwidth; validate in your environment.

Metadata Backup Strategy

• KRaft (3.5+): kafka-metadata-quorum snapshot captures controller quorum state; run on the controller leader.
• ZooKeeper: zkCli.sh get /cluster/meta + /brokers + /config exports; requires ensemble quorum.

Topic Selection Rules

• Always include: __consumer_offsets, __transaction_state, _schemas (Schema Registry), __consumer_timestamps.
• Include __cluster_metadata (KRaft) only via quorum snapshot, not file copy.
• Exclude other __* internal topics unless performing full cluster disaster recovery.

Consistency Guarantees

• Crash-consistent: Volume snapshots without quiesce; broker recovers from last checkpoint (risk: partial segment loss).
• Application-consistent: Pause producers (kafka-producer-perf-test --pause), wait for consumer lag=0, flush (producer.flush()), then snapshot. Adds 30–120s downtime but guarantees offset/transaction integrity.

Version & Environment Inventory

Run all discovery commands read-only; capture output to a timestamped inventory file (inventory-$(date +%Y%m%d-%H%M%S).txt).

# Version & protocol compatibility
kafka-broker-api-versions --bootstrap-server $BOOTSTRAP --command-config $CLIENT_PROPS

# KRaft quorum health (3.5+)
kafka-metadata-quorum --bootstrap-server $BOOTSTRAP --command-config $CLIENT_PROPS describe --status

# Topic configurations, replication factor, cleanup.policy
kafka-topics --bootstrap-server $BOOTSTRAP --command-config $CLIENT_PROPS --describe

# Log directory sizes, offline directories
kafka-log-dirs --bootstrap-server $BOOTSTRAP --command-config $CLIENT_PROPS --describe --topic-list "$TOPICS"

# ZooKeeper metadata (if applicable)
zookeeper-shell $ZK_HOST get /cluster/meta 2>/dev/null
zookeeper-shell $ZK_HOST ls /brokers/ids 2>/dev/null

Verification Signals

• Broker count from kafka-broker-api-versions matches expected cluster size.
• KRaft: LeaderId matches one voter; CurrentVoters count ≥ 3; HighWatermark advancing.
• ZooKeeper: Ensemble size odd (≥3); /cluster/meta returns valid JSON.
• No offline log directories (OfflineLogDirectoryCount = 0).

Backup Execution Procedures

Method A: MirrorMaker 2 (Active-Passive DR)

Prerequisites

• Target cluster reachable from MM2 Connect workers.
• MM2 Connect cluster deployed (separate from source/target brokers).
• offset-syncs.topic.replication.factor=3, heartbeats.topic.replication.factor=3, checkpoints.topic.replication.factor=3.
• Source and target clusters on compatible versions (MM2 since 2.4; 3.5+ recommended).

Connector Configuration (mm2-dr-connector.json)

{
  "name": "mm2-dr-prod-to-dr",
  "config": {
    "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
    "source.cluster.alias": "prod",
    "target.cluster.alias": "dr",
    "source.cluster.bootstrap.servers": "$PROD_BOOTSTRAP",
    "target.cluster.bootstrap.servers": "$DR_BOOTSTRAP",
    "source.cluster.security.protocol": "SASL_SSL",
    "target.cluster.security.protocol": "SASL_SSL",
    "source.cluster.sasl.mechanism": "SCRAM-SHA-512",
    "target.cluster.sasl.mechanism": "SCRAM-SHA-512",
    "topics": ".*",
    "groups": ".*",
    "emit.checkpoints.interval.seconds": "10",
    "sync.topic.acls.enabled": "false",
    "sync.group.offsets.enabled": "true",
    "sync.group.offsets.interval.seconds": "60",
    "offset-syncs.topic.replication.factor": "3",
    "heartbeats.topic.replication.factor": "3",
    "checkpoints.topic.replication.factor": "3",
    "replication.factor": "3",
    "tasks.max": "4"
  }
}

Deploy & Verify

# Deploy via Connect REST
curl -X POST -H "Content-Type: application/json" --data @mm2-dr-connector.json $CONNECT_URL/connectors

# Verify offset sync lag = 0
kafka-consumer-groups --bootstrap-server $DR_BOOTSTRAP --command-config $DR_PROPS \
 --group mm2-offset-syncs.prod --describe

✅ Verification Signal: LAG column = 0 for all partitions; CURRENT-OFFSET matches source LOG-END-OFFSET within 10s.

Method B: Kafka Connect Replicator (Confluent Platform)

Prerequisites

• Confluent Platform license.
• Schema Registry on both clusters with compatible subject compatibility mode.
• offset.topic.replication.factor=3, config.topic.replication.factor=3, status.topic.replication.factor=3.

Connector Configuration (replicator-dr.json)

{
  "name": "replicator-prod-to-dr",
  "config": {
    "connector.class": "io.confluent.connect.replicator.ReplicatorSourceConnector",
    "topic.rename.format": "${topic}",
    "src.kafka.bootstrap.servers": "$PROD_BOOTSTRAP",
    "dest.kafka.bootstrap.servers": "$DR_BOOTSTRAP",
    "src.kafka.security.protocol": "SASL_SSL",
    "dest.kafka.security.protocol": "SASL_SSL",
    "src.kafka.sasl.mechanism": "SCRAM-SHA-512",
    "dest.kafka.sasl.mechanism": "SCRAM-SHA-512",
    "src.consumer.group.id": "replicator-prod-to-dr",
    "offset.topic.replication.factor": "3",
    "config.topic.replication.factor": "3",
    "status.topic.replication.factor": "3",
    "topic.regex": ".*",
    "tasks.max": "4"
  }
}

Method C: Volume/Block Snapshots

Prerequisites

• Broker stopped OR filesystem quiesce + controller write pause.
• For KRaft: pause controller metadata writes before snapshot.
• IAM permissions: ec2:CreateSnapshot, ec2:CreateTags, ec2:DescribeVolumes.

KRaft Controller Quiesce (3.5+)

# Trigger preferred leader election for __cluster_metadata partition 0 to drain pending writes
kafka-leader-election --bootstrap-server $BOOTSTRAP --command-config $CLIENT_PROPS \
 --election-type PREFERRED --topic __cluster_metadata --partition 0

# Wait for election to complete (monitor ActiveControllerCount=1)

EBS Snapshot with Lifecycle Tags

aws ec2 create-snapshot \
 --volume-id $VOL_ID \
 --description "kafka-broker-$BROKER_ID-$(date +%s)" \
 --tag-specifications 'ResourceType=snapshot,Tags=[{Key=KafkaBackup,Value=true},{Key=BrokerId,Value='$BROKER_ID'},{Key=Cluster,Value='$CLUSTER_NAME'},{Key=Environment,Value=production}]'

Restore Procedure

1. Attach restored volume to replacement instance at same device path (/dev/xvdf → /var/lib/kafka/data).
2. Critical: Fix broker.id mismatch before start.

• ZooKeeper: Edit meta.properties in log.dir: broker.id=$ORIGINAL_BROKER_ID.
• KRaft: Update controller.quorum.voters in server.properties or dynamic config to include restored broker ID.

1. Start broker; verify kafka-broker-api-versions and ISR recovery (UnderReplicatedPartitions=0).

Method D: Tiered Storage Export (Kafka 3.6+)

Prerequisites

• remote.log.storage.system.enable=true.
• remote.storage.s3.bucket=$BUCKET (or GCS/Azure equivalent).
• remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager.
• Broker has s3:GetObject, s3:PutObject on bucket.

Export Command

Verify syntax in target version docs; CLI is evolving.

kafka-tiered-storage --bootstrap-server $BOOTSTRAP --command-config $CLIENT_PROPS \
 --export --topic $TOPIC --partition $PART \
 --remote-path s3://$BUCKET/export/$TOPIC-$PART-$(date +%s)

Import & Validate

kafka-tiered-storage --bootstrap-server $TARGET_BOOTSTRAP --command-config $TARGET_PROPS \
 --import --remote-path s3://$BUCKET/export/$TOPIC-$PART-<timestamp> \
 --local-log-dir /var/lib/kafka/data

# Validate index integrity
kafka-dump-log --files /var/lib/kafka/data/$TOPIC-$PART/00000000000000000000.log --verify-index-only

✅ Verification Signal: kafka-dump-log --verify-index-only exits 0 with no "Corrupt index" errors.

Method E: kafka-dump-log (Segment Inspection & Point-in-Time Recovery)

Use Cases: Segment corruption diagnosis, forensic recovery of deleted segments, checksum baseline.

# Deep iteration dump with data records (for checksum)
kafka-dump-log --files /var/lib/kafka/data/$TOPIC-$PART/00000000000000000000.log \
 --print-data-log --deep-iteration > /backup/$TOPIC-$PART-$(date +%s).dump

# Checksum pipeline (strip header lines)
kafka-dump-log --files /var/lib/kafka/data/$TOPIC-$PART/00000000000000000000.log \
 --deep-iteration | grep -v "^Dumping" | sha256sum

Verification & Diagnostics

Run validation after every backup or restore operation. Automate in CI/CD or cron with alerting on failure.

Post-Backup Validation Commands

Consumer Group Offset Validation

# Export source offsets
kafka-consumer-groups --bootstrap-server $SRC --command-config $SRC_PROPS \
 --group $CONSUMER_GROUP --export > /backup/offsets-$CONSUMER_GROUP-$(date +%s).json

# Import to target (dry-run first with --dry-run)
kafka-consumer-groups --bootstrap-server $TGT --command-config $TGT_PROPS \
 --import --input-file /backup/offsets-$CONSUMER_GROUP.json --reset-offsets --execute

✅ Verification Signal: Target consumer group shows LAG stable (not growing), no UNKNOWN_MEMBER_ID members.

Transaction Log Integrity

# Count control records (commit/abort markers) in __transaction_state
kafka-dump-log --files /var/lib/kafka/data/__transaction_state-0/00000000000000000000.log \
 --print-data-log --deep-iteration | grep -c "control record"

Compare count between source and restored replica; mismatch indicates transaction loss.

Monitoring Alerts for Backup Health

• kafka_server_BrokerTopicMetrics_MessagesInPerSec drop > 50% during backup window → investigate throttle.
• kafka_controller_KafkaController_ActiveControllerCount != 1 → controller instability.
• kafka_log_LogManager_OfflineLogDirectoryCount > 0 → disk failure during snapshot.
• Custom: MM2 mm2-source-task-lag-max > 10000 for 5m → replication stall.

Failure Modes & Recovery

MM2 Connector Task FAILED

Symptom: Task status FAILED in connect-statuses topic; logs show TimeoutException or AuthenticationException.

Diagnosis:

# Check task status
curl $CONNECT_URL/connectors/mm2-dr-prod-to-dr/status | jq '.tasks[] | select(.state=="FAILED")'

Recovery:

# Restart specific task
curl -X POST $CONNECT_URL/connectors/mm2-dr-prod-to-dr/tasks/0/restart
# If persistent, check source/target connectivity, ACLs, and offset-syncs topic replication

Snapshot Restore: Broker ID Mismatch

Symptom: Broker starts but cannot join cluster; logs show "Broker ID X already registered" or KRaft voter mismatch.

Recovery:

• ZooKeeper: Stop broker, edit meta.properties in log.dir to original broker.id, restart.
• KRaft: Update controller.quorum.voters in server.properties on all controllers to include restored broker ID; rolling restart controllers.

ISR Shrink After Restore

Symptom: UnderReplicatedPartitions > 0 persists > 5 min after broker restart.

Recovery:

# Trigger preferred leader election for all partitions
kafka-leader-election --bootstrap-server $BOOTSTRAP --command-config $CLIENT_PROPS \
 --election-type PREFERRED --all-topic-partitions

# Monitor until UnderReplicatedPartitions=0
watch -n 10 "kafka-server-metrics --bootstrap-server $BOOTSTRAP --command-config $CLIENT_PROPS | grep UnderReplicatedPartitions"

Offset Reset Storm

Symptom: Consumers reprocess from earliest after restore; lag spikes, duplicate processing.

Recovery:

# Pause consumers (deploy config change or SIGSTOP)
# Reset offsets to committed position (not earliest)
kafka-consumer-groups --bootstrap-server $TARGET --command-config $TARGET_PROPS \
 --group $GROUP --reset-offsets --to-current --execute --dry-run
# Verify, then execute without --dry-run
# Resume consumers

Schema Registry Incompatibility

Symptom: Consumers fail with SchemaParseException or IncompatibleSchemaException after DR cutover.

Recovery:

# Export all subjects from source SR
sr-cli subjects | xargs -I{} sr-cli get-schema {} > /backup/sr-subjects-$(date +%s).json

# Import to target SR (verify compatibility mode first)
cat /backup/sr-subjects.json | jq -c '.[]' | while read subject; do
 sr-cli register --subject "$(echo $subject | jq -r .subject)" --schema "$(echo $subject | jq -r .schema)"
done

Rollback Procedures Per Method

Operations Checklist

• [ ] Inventory captured & version-verified (Section 3 commands executed, output archived).
• [ ] Backup method selected per RPO/RTO matrix (Section 2 decision documented).
• [ ] Prerequisites met: storage headroom (1.5×–2×), network bandwidth, IAM roles, Schema Registry access.
• [ ] Test restore executed in staging within last 30 days (documented runbook with timestamps).
• [ ] Verification scripts automated: checksum comparison, consumer lag, ISR health, transaction log count.
• [ ] Runbook documented with blast radius, rollback steps, on-call contacts, communication plan.
• [ ] Secrets managed via Vault/Secrets Manager; no plaintext in configs or command history.
• [ ] Monitoring alerts tuned: backup job success/failure, replication lag, controller health, offline log dirs.
• [ ] KRaft metadata quorum snapshot scheduled daily (kafka-metadata-quorum snapshot --output-file /backup/kraft-snapshot-$(date +%s).bin).
• [ ] Schema Registry subject export/import tested quarterly.

Conclusion

Production Kafka backup is not a single tool but a portfolio of version-scoped strategies matched to explicit RPO/RTO targets. MirrorMaker 2 provides near-zero RPO for active-active and active-passive topologies but demands operational maturity; volume snapshots offer simplicity with hour-scale RPO; tiered storage export (3.6+) bridges the gap for selective recovery. KRaft metadata quorum snapshots (KIP-848) and __consumer_offsets / __transaction_state inclusion are non-negotiable for consistent restore. Every method requires automated verification—checksum match, lag=0, ISR stable, transaction log integrity—before declaring a backup valid. Test restores in staging monthly, document rollback triggers, and keep secrets out of command history. The only backup that matters is the one you have successfully restored.