Kafka monitoring and alerts with practical examples

Intro

Kafka monitoring is useful only when it helps an operator answer three questions quickly: what is broken, who is affected, and what action is safe right now. A dashboard full of broker CPU charts is not enough. You need broker health metrics, topic and partition signals, consumer lag, request latency, and alert rules that point to a specific response.

This guide assumes a production-like Apache Kafka cluster monitored through JMX, Prometheus, and Grafana. The examples use placeholders such as <bootstrap-server> and <consumer-group>. Replace them with your environment values, but do not paste credentials, tokens, or private hostnames into shared runbooks.

The focus is practical: real JMX metric names, alert thresholds you can start with, command-line checks, and an incident scenario where a consumer group falls behind during a broker restart. Thresholds should always be adjusted after you observe normal traffic patterns, but the examples below are intentionally concrete so that teams can implement a useful first version instead of debating abstract monitoring categories.

Version and Environment Inventory

Before changing alerts, capture the Kafka version, deployment topology, exporter source, and consumer groups that matter to the business. Kafka metrics vary by broker version, exporter configuration, and whether you run ZooKeeper-based Kafka or KRaft mode.

Start with read-only checks:

kafka-broker-api-versions.sh --bootstrap-server <bootstrap-server>:9092

Confirm the broker list and controller state:

kafka-metadata-quorum.sh --bootstrap-server <bootstrap-server>:9092 describe --status

If the cluster still uses ZooKeeper, the KRaft command may not apply. In that case, document the ZooKeeper ensemble separately and confirm which broker is controller through JMX or your platform tooling.

Record these inventory items:

• Kafka version and whether the cluster uses KRaft or ZooKeeper.
• Number of brokers, racks or availability zones, and replication factor for critical topics.
• Minimum in-sync replica settings for producer-critical topics, especially min.insync.replicas.
• Monitoring path: JMX exporter on each broker, Kafka exporter for consumer lag, or both.
• Critical consumer groups, their owning service, expected peak throughput, and acceptable lag window.

Useful broker JMX object names to confirm are present:

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
kafka.server:type=ReplicaManager,name=OfflineReplicaCount
kafka.controller:type=KafkaController,name=ActiveControllerCount
kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer

For Java consumers that expose JMX, capture these client-side metrics as well:

kafka.consumer:type=consumer-fetch-manager-metrics,client-id=<client-id>,name=records-lag-max
kafka.consumer:type=consumer-fetch-manager-metrics,client-id=<client-id>,name=records-consumed-rate
kafka.consumer:type=consumer-coordinator-metrics,client-id=<client-id>,name=commit-rate
kafka.consumer:type=consumer-coordinator-metrics,client-id=<client-id>,name=rebalance-rate-per-hour

The inventory is not paperwork. It prevents false alerts, identifies missing exporters, and tells the on-call engineer whether a lag spike affects a batch reporting service or a payment authorization path.

Safe Configuration Path

Add monitoring in layers. First expose metrics, then build dashboards, then enable alerts. Avoid enabling a large alert pack without confirming metric names and normal baselines.

A common Prometheus setup uses JMX exporter for broker metrics. Depending on your exporter rules, JMX names may appear as Prometheus metrics such as:

kafka_server_replicamanager_underreplicatedpartitions
kafka_server_replicamanager_offlinereplicacount
kafka_controller_kafkacontroller_activecontrollercount
kafka_network_requestmetrics_totaltimems
kafka_server_brokertopicmetrics_messagesin_total

Validate that Prometheus is scraping every broker:

curl -s http://<prometheus-host>:9090/api/v1/targets | grep -E "kafka|broker"

Then query a key metric directly:

curl -G http://<prometheus-host>:9090/api/v1/query \
  --data-urlencode 'query=sum(kafka_server_replicamanager_underreplicatedpartitions)'

For consumer lag, many teams use Kafka exporter metrics similar to:

kafka_consumergroup_lag
kafka_consumergroup_current_offset
kafka_topic_partition_current_offset

Confirm lag is visible for a known group:

curl -G http://<prometheus-host>:9090/api/v1/query \
  --data-urlencode 'query=sum by (consumergroup, topic) (kafka_consumergroup_lag{consumergroup="<consumer-group>"})'

A safe initial Grafana dashboard should have these panels:

• Under-replicated partitions: sum(kafka_server_replicamanager_underreplicatedpartitions)
• Offline replicas: sum(kafka_server_replicamanager_offlinereplicacount)
• Active controllers: sum(kafka_controller_kafkacontroller_activecontrollercount)
• Produce request latency p95 or p99 from request metrics, if histogram or summary buckets are available.
• Bytes in and bytes out per broker.
• Consumer lag by consumer group and topic.
• Consumer lag trend using deriv(kafka_consumergroup_lag[10m]).

Keep the first version small. A dashboard that highlights replication, controller health, traffic, latency, and lag is more useful during an incident than twenty panels with no response path.

Verification and Diagnostics

Use alert rules that are specific enough to reduce noise but sensitive enough to catch real degradation. The following Prometheus rules are a practical starting point for a medium-sized production cluster. They can also be used as Grafana-managed alert expressions if your Grafana instance evaluates Prometheus queries.

groups:
  - name: kafka-alerts
    rules:
      - alert: KafkaUnderReplicatedPartitions
        expr: sum(kafka_server_replicamanager_underreplicatedpartitions) > 0
        for: 10m
        labels:
          severity: warning
          service: kafka
        annotations:
          summary: "Kafka has under-replicated partitions"
          description: "Under-replicated partitions have been above 0 for 10 minutes. Check broker health, ISR shrink events, and network saturation."

      - alert: KafkaOfflineReplicas
        expr: sum(kafka_server_replicamanager_offlinereplicacount) > 0
        for: 2m
        labels:
          severity: critical
          service: kafka
        annotations:
          summary: "Kafka has offline replicas"
          description: "One or more replicas are offline. Data availability may be reduced for affected partitions."

      - alert: KafkaControllerCountInvalid
        expr: sum(kafka_controller_kafkacontroller_activecontrollercount) != 1
        for: 5m
        labels:
          severity: critical
          service: kafka
        annotations:
          summary: "Kafka active controller count is not 1"
          description: "Expected exactly one active controller across the cluster. Investigate broker or quorum instability."

      - alert: KafkaConsumerGroupLagHigh
        expr: |
          sum by (consumergroup, topic) (
            kafka_consumergroup_lag{consumergroup!~"console-consumer-.*"}
          ) > 50000
          and
          sum by (consumergroup, topic) (
            deriv(kafka_consumergroup_lag{consumergroup!~"console-consumer-.*"}[10m])
          ) > 0
        for: 10m
        labels:
          severity: warning
          service: kafka
        annotations:
          summary: "Kafka consumer group lag is high and increasing"
          description: "Consumer group {{ $labels.consumergroup }} on topic {{ $labels.topic }} has lag above 50000 and is still falling behind."

      - alert: KafkaConsumerGroupLagCritical
        expr: |
          sum by (consumergroup, topic) (
            kafka_consumergroup_lag{consumergroup!~"console-consumer-.*"}
          ) > 250000
        for: 5m
        labels:
          severity: critical
          service: kafka
        annotations:
          summary: "Kafka consumer group lag is critical"
          description: "Consumer group {{ $labels.consumergroup }} on topic {{ $labels.topic }} has lag above 250000 for 5 minutes."

Consumer lag thresholds should reflect time, not just message count. If a topic normally processes 10,000 messages per second, a lag of 50,000 is only five seconds. If another topic processes 100 messages per second, the same lag is more than eight minutes. Use message thresholds at first, then refine with topic-specific thresholds once you know normal throughput.

During diagnostics, compare Prometheus lag with Kafka CLI output:

kafka-consumer-groups.sh \
  --bootstrap-server <bootstrap-server>:9092 \
  --describe \
  --group <consumer-group>

Example output:

GROUP           TOPIC            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG     CONSUMER-ID  HOST        CLIENT-ID
payments-api    payments.events  0          8934201         8979201         45000   consumer-1   /10.0.4.21  payments-1
payments-api    payments.events  1          7721009         7811009         90000   consumer-2   /10.0.4.22  payments-2

If lag is high but not increasing, the consumers may already be recovering. If lag is high and increasing, check consumer errors, downstream database latency, rebalance activity, and whether enough partitions exist to scale the group.

Failure Modes and Recovery

A realistic incident: a three-broker cluster is undergoing a rolling restart. Broker broker-2 restarts slower than expected after a package update. Producers continue writing to payments.events, but the payments-api consumer group begins falling behind.

Alerts show:

WARNING KafkaUnderReplicatedPartitions
sum(kafka_server_replicamanager_underreplicatedpartitions) = 18 for 10m

WARNING KafkaConsumerGroupLagHigh
consumergroup="payments-api", topic="payments.events"
lag = 132000, deriv over 10m > 0

CRITICAL KafkaConsumerGroupLagCritical
consumergroup="payments-api", topic="payments.events"
lag = 278000 for 5m

First response:

1. Stop the rolling restart. Do not restart another broker while partitions are under-replicated.
2. Confirm broker health and cluster visibility:

kafka-broker-api-versions.sh --bootstrap-server <bootstrap-server>:9092

1. List under-replicated partitions:

kafka-topics.sh \
  --bootstrap-server <bootstrap-server>:9092 \
  --describe \
  --under-replicated-partitions

1. Check the affected consumer group:

kafka-consumer-groups.sh \
  --bootstrap-server <bootstrap-server>:9092 \
  --describe \
  --group payments-api

1. Check whether the consumer group is rebalancing repeatedly. If client JMX is available, inspect rebalance-rate-per-hour and commit-rate. A low or zero commit rate while lag grows usually means consumers are stuck, crashing, or blocked by a downstream dependency.

Safe recovery decisions:

• If broker broker-2 is still starting, wait for it to rejoin before making partition assignment changes.
• If consumers are healthy but under-provisioned, scale the consumer deployment only up to the number of partitions for the topic.

kubectl scale deployment payments-api-consumer \
  --namespace <namespace> \
  --replicas=8

• If consumers are failing because a downstream database is slow, scaling may make the outage worse. Throttle intake or fix the downstream dependency first.
• Do not reset offsets to skip messages unless the application owner explicitly accepts data loss or replay semantics are understood.

Recovery is verified when these signals are true:

sum(kafka_server_replicamanager_underreplicatedpartitions) = 0
sum(kafka_server_replicamanager_offlinereplicacount) = 0
sum by (consumergroup, topic) (deriv(kafka_consumergroup_lag[10m])) < 0 for payments-api/payments.events
KafkaConsumerGroupLagCritical is resolved

After recovery, document the restart delay, the maximum lag, time to catch up, whether consumers were scaled, and whether restart procedures need a longer pause between brokers.

Operations Checklist

Use this checklist for daily operations and incident readiness:

• Confirm every broker is scraped by Prometheus and visible in Grafana.
• Alert when under-replicated partitions stay above 0 for 10 minutes.
• Alert immediately when offline replicas remain above 0 for 2 minutes.
• Alert when active controller count is not exactly 1 for 5 minutes.
• Monitor consumer lag by consumergroup and topic, not only as a cluster total.
• Add separate warning and critical lag thresholds for business-critical groups.
• Track lag direction with deriv(kafka_consumergroup_lag[10m]); high but falling lag is different from high and rising lag.
• Keep a CLI diagnostic command for each alert in the runbook.
• During broker maintenance, pause the rollout if under-replicated partitions appear and do not clear.
• Verify recovery with metrics, not just with a successful service restart.

For dashboards, prefer panels that answer operational questions. For example, show top 10 consumer groups by lag, top topics by bytes in, request latency by broker, and under-replicated partitions as a single large stat panel. Put critical panels at the top so an on-call engineer can understand impact in less than a minute.

Conclusion

Kafka monitoring and alerts are effective when they connect metrics to decisions. The most important signals are replication health, controller health, broker request latency, throughput, and consumer lag by group and topic. Generic CPU and memory charts can support diagnosis, but they should not be the only alert source.

Start with concrete JMX metrics, verify the Prometheus names produced by your exporters, and use alert rules with explicit thresholds such as under-replicated partitions above 0 for 10 minutes or consumer lag above 50,000 and still increasing. Then tune those values against real traffic and business tolerance.

The best Kafka incident response is calm and observable: stop risky changes, confirm broker and partition state, compare lag from Prometheus with Kafka CLI output, scale consumers only when it is safe, and verify that lag is decreasing before closing the incident.