Kafka Security Hardening with Practical Examples

Apache Kafka often sits at the center of event-driven architectures, making it a high-value target for attackers. Hardening Kafka is not about enabling every security feature simultaneously. It is a disciplined process of applying the right controls in the correct sequence, verifying each change, and maintaining a clean rollback path. This guide walks through a practical, step-by-step hardening workflow: inventory, TLS encryption, SASL/SCRAM authentication, least-privilege ACLs, network reduction, metadata plane protection, and safe cutover. Every example is constructed to be clear, auditable, and adaptable to your environment.

Version and Environment Inventory

Before changing any configuration, capture a concise inventory. This baseline determines which controls you enable and in what order, preventing the most common surprises: protocol mismatches, hostname verification failures, and ACL denials for internal components.

• Kafka version and mode: Note whether you run Kafka 3.x in ZooKeeper mode or KRaft (controller quorum) mode. KRaft consolidates metadata and changes controller listener configuration.
• Java runtime and OS: Record JDK vendor and patch level (JDK 11+ recommended), Linux distribution, and kernel version.
• Cluster topology: Document the number of brokers, hostnames, IPs, data directories, and rack awareness settings.
• Current listeners and ports: List existing PLAINTEXT, SSL, and SASL_SSL listeners with their ports.
• Client inventory: Identify applications, languages, and client library versions (Java, Python, Go, etc.). Map which topics and consumer groups each client uses.
• Certificate authority and trust roots: Specify internal CA or public CA, keystore/truststore format (PKCS12 or JKS), key lengths, and algorithms.
• Networking: Note firewalls, load balancers, DNS configuration, TLS termination points, internal versus external segments, NAT rules, and egress policies.

Capture this inventory once and keep it accessible during the rollout.

Safe Configuration Path: Incremental Hardening

Harden in small, verifiable steps. Keep a working, known-good configuration path during each step and remove it only after verification.

Step 0: Choose a Narrow Pilot

Select one broker and one test client (a producer and a consumer). Create a canary topic such as sec.pilot.events. This pilot must be measurable and easy to inspect locally before any cluster-wide rollout.

Step 1: Add a TLS Listener (Keep PLAINTEXT Active)

Goal: Enable encrypted traffic without breaking existing clients.

1. Obtain certificates Use your internal CA. Ensure certificates include the broker's DNS name in the Subject Alternative Name (SAN) extension. Store files under /etc/kafka/ssl/ with permissions 600 and owner kafka.

2. Configure the broker Add a new SSL listener while retaining the PLAINTEXT listener. Update server.properties:

# Add new listeners
listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093
advertised.listeners=PLAINTEXT://broker1.example.com:9092,SSL://broker1.example.com:9093
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL
inter.broker.listener.name=SSL

# TLS settings
ssl.keystore.location=/etc/kafka/ssl/broker1.keystore.p12
ssl.keystore.password=changeMeKeystore
ssl.keystore.type=PKCS12
ssl.truststore.location=/etc/kafka/ssl/broker1.truststore.p12
ssl.truststore.password=changeMeTrust
ssl.truststore.type=PKCS12
ssl.client.auth=required

# Authorizer will be enabled later
allow.everyone.if.no.acl.found=true

3. Restart and verify Restart the broker. Confirm it starts without errors and the PLAINTEXT listener remains available. Verify the TLS handshake from an admin host that trusts your CA:

openssl s_client -connect broker1.example.com:9093 -servername broker1.example.com -tls1_2 -brief

Expected output: Verification: OK and a negotiated TLS version and cipher. If you see verify error: num=20: unable to get local issuer certificate, your truststore or certificate chain is incorrect.

Step 2: Require Client Authentication with SASL/SCRAM (New Port)

Goal: Add authenticated access on a separate listener so you can test without affecting the TLS-only path.

1. Add SASL_SSL listener Update server.properties:

# Add a SASL_SSL listener
listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093,SASL_SSL://0.0.0.0:9094
advertised.listeners=PLAINTEXT://broker1.example.com:9092,SSL://broker1.example.com:9093,SASL_SSL://broker1.example.com:9094
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_SSL:SASL_SSL
inter.broker.listener.name=SSL

# Enable SASL SCRAM
sasl.enabled.mechanisms=SCRAM-SHA-256,SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512

# Prepare to enable ACLs later
authorizer.class.name=org.apache.kafka.server.authorizer.StandardAuthorizer
allow.everyone.if.no.acl.found=true
super.users=User:admin

2. Provide JAAS login for the broker Create /etc/kafka/kafka_server_jaas.conf:

KafkaServer {
    org.apache.kafka.common.security.scram.ScramLoginModule required
    username="broker"
    password="changeMeBroker!";
};

Set the environment variable for the broker service:

export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf"

3. Create service accounts with SCRAM credentials From an admin machine with access to port 9094:

# Writer
kafka-configs.sh --bootstrap-server broker1.example.com:9094 \
  --alter --add-config 'SCRAM-SHA-512=[password=changeMeNow!]' \
  --entity-type users --entity-name app_writer

# Reader
kafka-configs.sh --bootstrap-server broker1.example.com:9094 \
  --alter --add-config 'SCRAM-SHA-512=[password=changeMeRead!]' \
  --entity-type users --entity-name app_reader

4. Test authentication from a client Create client-writer.properties:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
ssl.truststore.location=/etc/kafka/ssl/ca.truststore.p12
ssl.truststore.password=changeMeTrust
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="app_writer" password="changeMeNow!";

Produce to the canary topic:

kafka-topics.sh --bootstrap-server broker1.example.com:9094 --create --topic sec.pilot.events --partitions 1 --replication-factor 1
echo 'hello' | kafka-console-producer.sh --bootstrap-server broker1.example.com:9094 \
  --producer.config client-writer.properties --topic sec.pilot.events

Expected result: message accepted. If you see SASL authentication failed, verify the username/password and confirm the client trusts the broker's certificate.

Step 3: Enforce Least-Privilege with Kafka ACLs

Goal: Deny by default and allow only what a principal needs.

1. Switch to deny-by-default In server.properties:

allow.everyone.if.no.acl.found=false

Restart the broker.

2. Grant minimal rights to service accounts

# Allow writer to create and write to the topic
kafka-acls.sh --bootstrap-server broker1.example.com:9094 \
  --add --allow-principal User:app_writer --operation Create --topic sec.pilot.events
kafka-acls.sh --bootstrap-server broker1.example.com:9094 \
  --add --allow-principal User:app_writer --operation Write --topic sec.pilot.events
kafka-acls.sh --bootstrap-server broker1.example.com:9094 \
  --add --allow-principal User:app_writer --operation IdempotentWrite --cluster

# Allow reader to read the topic and join a consumer group
kafka-acls.sh --bootstrap-server broker1.example.com:9094 \
  --add --allow-principal User:app_reader --operation Read --topic sec.pilot.events
kafka-acls.sh --bootstrap-server broker1.example.com:9094 \
  --add --allow-principal User:app_reader --operation Read --group sec.pilot.readers

3. Verify ACLs List ACLs for the topic:

kafka-acls.sh --bootstrap-server broker1.example.com:9094 --list --topic sec.pilot.events

Attempt an unauthorized action: try to consume with app_writer and expect a denial in the client and a ResourceAuthorizationException log entry on the broker.

Step 4: Reduce Network Exposure

• Bind listeners only to required interfaces. Prefer binding to internal IPs explicitly, or use 0.0.0.0 with firewall restrictions.
• Do not expose PLAINTEXT outside a controlled admin network.
• Restrict access to controller listeners (KRaft) and ZooKeeper (if used) to brokers and admins only.

Quick reference for typical listeners and ports:

Step 5: Secure the Metadata Layer

ZooKeeper mode:

• Set zookeeper.set.acl=true in server.properties.
• Enable ZooKeeper TLS client: zookeeper.client.secure=true, zookeeper.ssl.client.enable=true and provide keystore/truststore.
• Restrict ZooKeeper to broker/admin subnets.

KRaft mode:

• Define a dedicated controller listener with TLS, e.g., listener.security.protocol.map includes CONTROLLER:SSL and controller.listener.names=CONTROLLER.
• Restrict the controller port to brokers and controllers only.

Step 6: Plan the Cutover and Deprecate PLAINTEXT

• Migrate a subset of real clients to SASL_SSL with ACLs.
• Observe for 24–72 hours.
• If stable, remove PLAINTEXT from listeners and advertised.listeners and restart brokers during a maintenance window.

Verification and Diagnostics

Make each control observable with concrete tests.

TLS Verification

• Handshake and chain:

  openssl s_client -connect broker1.example.com:9093 -servername broker1.example.com -tls1_2 -brief

Expected: Verification: OK and a modern cipher. If hostname mismatch appears, fix the certificate SANs.

• Client connection: Confirm connections succeed with security.protocol=SSL and fail when using PLAINTEXT on a restricted port.

SASL/SCRAM Verification

• Authentication success:

  kafka-console-producer.sh --bootstrap-server broker1.example.com:9094 \
    --producer.config client-writer.properties --topic sec.pilot.events

Type a few lines and press Ctrl-D. Expected: broker logs contain an entry similar to Authenticated principal=User:app_writer via SASL mechanism SCRAM-SHA-512 on listener SASL_SSL.

• Authentication failure: Change the password in client-writer.properties and confirm the client reports SASL authentication failed and the broker logs a failed authentication with your source IP.

ACL Enforcement

• Positive test: app_reader consumes successfully only from authorized topics and groups.

  kafka-console-consumer.sh --bootstrap-server broker1.example.com:9094 \
    --consumer.config client-reader.properties --topic sec.pilot.events \
    --group sec.pilot.readers --from-beginning --timeout-ms 5000

Expected: messages appear, then consumer exits on timeout.

• Negative test: Using app_reader, try producing to the topic and expect an authorization error.

Configuration Consistency

• Check inter.broker.listener.name and listener alignment on each node. All brokers must agree on the inter-broker listener and support its protocol.
• Confirm controller or ZooKeeper endpoints are reachable only from brokers.

Logging and Metrics Hygiene

• Ensure logs include principal, client-id, and listener in authentication and authorization entries.
• Export broker request metrics for new listeners to spot spikes in authentication failures.

Failure Modes and Recovery

Knowing how things break lets you fix them fast.

TLS Failures

• Symptom: PKIX path building failed or certificate_unknown.
• Cause: Missing CA in client truststore or incomplete chain.
• Fix: Import the issuing CA and intermediates into the client truststore.
• Symptom: hostname verification failed.
• Cause: Certificate SAN does not match DNS.
• Fix: Reissue certs with proper DNS SANs. Verify with:

  openssl x509 -in broker1.crt -noout -text | grep -A1 'Subject Alternative Name'

• Symptom: Broker fails to start after enabling SSL.
• Cause: Wrong keystore password or file permissions.
• Fix: Correct passwords, set chmod 600 on keystore/truststore, ensure owner is kafka.

SASL/SCRAM Failures

• Symptom: Authentication failed for all clients.
• Cause: JAAS not loaded or mechanism mismatch.
• Fix: Set KAFKA_OPTS with JAAS path; confirm sasl.enabled.mechanisms includes your mechanism.
• Symptom: Only one user fails.
• Cause: Missing or wrong SCRAM credential.
• Fix: Reapply credential:

  kafka-configs.sh --bootstrap-server broker1.example.com:9094 \
    --alter --add-config 'SCRAM-SHA-512=[password=newSecret]' \
    --entity-type users --entity-name affected_user

ACL Denials

• Symptom: Producers or consumers suddenly stop with authorization errors.
• Cause: Deny-by-default without corresponding allow ACLs.
• Fix: Add minimal ACLs for topic and group. Verify with kafka-acls.sh --list.

Inter-Broker Protocol Drift

• Symptom: Replication stalls after changing listeners.
• Cause: inter.broker.listener.name set to a listener that not all brokers expose.
• Fix: Ensure all brokers have the same inter-broker listener and security protocol, then rolling restart.

Metadata Plane Exposure

• Symptom: Unexpected connections to ZooKeeper or KRaft controller ports.
• Cause: Firewall gaps.
• Fix: Restrict those ports to broker and admin segments only.

Rollback and Recovery

Keep rollback simple during the pilot.

• Preserve a working PLAINTEXT listener until after you verify SASL_SSL and ACLs. If SASL locks clients out, point clients back to PLAINTEXT as a short-term fallback while you fix credentials.
• Keep backups of server.properties, JAAS files, and keystores before each change.
• Revert ACLs if needed:

  kafka-acls.sh --bootstrap-server broker1.example.com:9094 \
    --remove --allow-principal User:app_writer --operation Write --topic sec.pilot.events

• Remove a user credential if created in error:

  kafka-configs.sh --bootstrap-server broker1.example.com:9094 \
    --alter --delete-config 'SCRAM-SHA-512' --entity-type users --entity-name app_writer

• Final fallback: restore prior config files and restart brokers in a controlled order.

Operations Checklist

Use this compact checklist during rollout and for periodic reviews.

Before Enabling New Controls

• [ ] Inventory Kafka version, mode (ZooKeeper vs KRaft), Java, OS.
• [ ] Confirm DNS and certificate SANs for all brokers.
• [ ] Prepare keystore/truststore with correct CA chain.
• [ ] Create pilot topic and select pilot clients.

TLS

• [ ] Add SSL listener on a new port and verify openssl s_client.
• [ ] Set ssl.client.auth=required and confirm mutual TLS if used.
• [ ] Bind or firewall listeners to internal networks.

SASL/SCRAM

• [ ] Enable sasl.enabled.mechanisms and set broker JAAS if required.
• [ ] Create per-service users with unique, rotated passwords.
• [ ] Test producer and consumer with security.protocol=SASL_SSL.

ACLs

• [ ] Switch to allow.everyone.if.no.acl.found=false only after test ACLs exist.
• [ ] Grant minimal topic, group, and cluster permissions.
• [ ] Verify both allowed and denied actions.

Metadata Layer

• [ ] Secure ZooKeeper or controller listeners with TLS and network ACLs.
• [ ] Set zookeeper.set.acl=true (ZooKeeper mode).

Decommission PLAINTEXT

• [ ] Migrate clients gradually; monitor auth failures.
• [ ] Remove PLAINTEXT listeners after a stable window.

Secrets and Files

• [ ] Keystore/truststore and JAAS permissions 600; owner kafka.
• [ ] No secrets in shell history or world-readable files.

Monitoring and Logs

• [ ] Alert on auth failures, TLS errors, and ACL denials.
• [ ] Periodically list ACLs and users; remove unused.

Practical Control Map

This table connects common security goals to specific Kafka controls.

Conclusion

Start with a small pilot that is easy to inspect and roll back. Add a TLS listener, introduce SASL/SCRAM on a new port, and flip on ACLs in deny-by-default mode only after accounts and rules are in place. Verify each step with concrete tests: TLS handshakes, authenticated principals, and ACL outcomes. As you expand to the full cluster, remove PLAINTEXT and tighten network access to broker, controller, and ZooKeeper ports. Extend the same principles to systems that integrate with Kafka: NiFi processors should use SASL_SSL and service-specific principals; Spark streaming and Structured Streaming jobs should authenticate and run with least-privilege ACLs; HDFS sinks and sources should trust the same CA and isolate principals; Airflow tasks that publish or consume should use per-task or per-connection credentials with narrow ACLs. With a measured, verifiable path and a clean rollback plan, you can harden Kafka without outages and with clear evidence that each control works as intended.