NiFi Monitoring and Alerts with Practical Examples

Intro

Monitoring Apache NiFi effectively means tracking data flow health, system resource usage, and component status to detect problems before they impact production pipelines. NiFi provides built-in monitoring capabilities through its API, reporting tasks, and integration with external systems like Prometheus, Grafana, and alerting tools.

This guide focuses on practical, actionable monitoring and alerting for NiFi. You'll learn how to:

• Collect key NiFi metrics (flow, processor, connection, JVM, and system metrics)
• Expose metrics to Prometheus and visualize them in Grafana
• Set up alerting rules and notification channels (email, Slack, PagerDuty)
• Diagnose common failure modes and recover from them
• Build an operational checklist for day-to-day monitoring

All examples assume a standard NiFi installation (version 1.13+), with commands that can be adapted to your environment. Replace placeholders like {NIFI_HOST}, {PORT}, {PROCESSOR_ID} with your actual values.

Version and Environment Inventory

Before setting up monitoring, you need to know exactly what you're running. Use the following read-only commands to gather version, configuration, and status information.

Check NiFi Version and Build Information

curl -k https://{NIFI_HOST}:{NIFI_PORT}/nifi-api/system-diagnostics | jq '.systemDiagnostics.aggregateSnapshot.versionInfo'

Expected output example:

{
  "niFiVersion": "1.16.3",
  "buildTag": "nifi-1.16.3-RC1",
  "buildTimestamp": "03/29/2022 13:47:50 UTC",
  "javaVendor": "Oracle Corporation",
  "javaVersion": "1.8.0_311"
}

If the command fails with a certificate error, use curl -k only if you understand and accept the security implications. In production, use proper trust stores.

Identify Cluster Topology

To see cluster nodes and their status:

curl -k https://{NIFI_HOST}:{NIFI_PORT}/nifi-api/controller/cluster | jq '.cluster.nodes[] | {nodeId, address, status}'

Expected output example:

{
  "nodeId": "node-1",
  "address": "nifi1.example.com:8443",
  "status": "CONNECTED"
}
{
  "nodeId": "node-2",
  "address": "nifi2.example.com:8443",
  "status": "CONNECTED"
}

Prerequisites Checklist

• Access to NiFi REST API (HTTPS usually enabled)
• jq installed for JSON parsing (sudo apt install jq or brew install jq)
• Network access to all NiFi nodes if clustered
• Read-only permissions (or a user with view access)

Read-Only Observation: System Diagnostics

curl -k https://{NIFI_HOST}:{NIFI_PORT}/nifi-api/system-diagnostics | jq '.systemDiagnostics.aggregateSnapshot'

This returns a wealth of system metrics: heap usage, non-heap usage, load average, disk usage, and more. Capture this baseline before making changes.

Capturing Current State for Change Control

Always record the current state before any change. For example, to list all reporting tasks:

curl -k https://{NIFI_HOST}:{NIFI_PORT}/nifi-api/controller/reporting-tasks | jq '.reportingTasks[] | {id, name, state, type}'

Expected output:

{
  "id": "reporting-task-1",
  "name": "AmbariReportingTask",
  "state": "RUNNING",
  "type": "org.apache.nifi.reporting.ambari.AmbariReportingTask"
}

Safe Configuration Path

NiFi's monitoring configuration involves enabling reporting tasks that send metrics to external systems. This section covers the safest way to enable Prometheus metrics, which is the foundation for many dashboards and alerts.

Enabling Prometheus Reporting Task

The Prometheus reporting task class is org.apache.nifi.reporting.prometheus.PrometheusReportingTask. It is included in NiFi 1.8+.

1. Identify the reporting task type

1. Create the reporting task via API (or use UI):

curl -k -X POST https://{NIFI_HOST}:{NIFI_PORT}/nifi-api/controller/reporting-tasks \
  -H 'Content-Type: application/json' \
  -d '{
    "revision": {"version": 0},
    "component": {
      "name": "PrometheusReportingTask",
      "type": "org.apache.nifi.reporting.prometheus.PrometheusReportingTask",
      "properties": {
        "prometheus-reporting-task-metrics-endpoint-port": "9092",
        "prometheus-reporting-task-metrics-endpoint-path": "/metrics"
      }
    }
  }'

Note: The exact property names may vary by version. Check the official NiFi documentation for your version. In NiFi 1.16, properties are:

• prometheus-reporting-task-metrics-endpoint-port
• prometheus-reporting-task-metrics-endpoint-path

After creation, get the task ID and start it:

1. Start the reporting task

# Get the task ID
TASK_ID=$(curl -k https://{NIFI_HOST}:{NIFI_PORT}/nifi-api/controller/reporting-tasks | jq -r '.reportingTasks[] | select(.name=="PrometheusReportingTask") | .id')

# Start the task
curl -k -X PUT https://{NIFI_HOST}:{NIFI_PORT}/nifi-api/reporting-tasks/$TASK_ID \
  -H 'Content-Type: application/json' \
  -d '{
    "revision": {"version": 1},
    "state": "RUNNING"
  }'

Hit the Prometheus endpoint from your monitoring server:

1. Verify the endpoint

curl http://{NIFI_HOST}:9092/metrics | head -20

Expected output includes metrics like:

nifi_amount_flowfiles_received{instance="...",} 0.0
nifi_amount_flowfiles_sent{instance="...",} 0.0
nifi_amount_flowfiles_queued{instance="...",} 10.0
nifi_jvm_heap_used{instance="...",} 3.2E8

Blast Radius and Security

• The Prometheus endpoint is unauthenticated by default. Bind it to a private network interface or protect it with a reverse proxy.
• Changing reporting task configurations can cause metrics interruptions; do it during a maintenance window if possible.
• Always keep a backup of the NiFi flow configuration (flow.xml.gz) before major changes.

Recovery Path

If the reporting task fails to start:

• Check logs at {NIFI_HOME}/logs/nifi-app.log for errors.
• Verify the configured port is not in use.
• If needed, stop and disable the task, then recreate with correct properties.

Verification and Diagnostics

After setting up monitoring, verify that metrics are flowing and dashboards are populated. This section shows diagnostic commands and what to look for.

Verify Prometheus Scraping

In your Prometheus server, check the target health:

curl http://{PROMETHEUS_HOST}:9090/api/v1/targets | jq '.data.activeTargets[] | {scrapeUrl, health, lastError}'

Expected output:

{
  "scrapeUrl": "http://nifi1.example.com:9092/metrics",
  "health": "up",
  "lastError": ""
}

If health is down, check network connectivity, firewall rules, and NiFi reporting task state.

Query Specific Metrics

To see queued flowfiles for a connection:

curl -s 'http://{PROMETHEUS_HOST}:9090/api/v1/query?query=nifi_amount_flowfiles_queued' | jq '.data.result[] | {instance: .metric.instance, value: .value[1]}'

Diagnostic Commands for NiFi Health

Check overall system health via NiFi API:

curl -k https://{NIFI_HOST}:{NIFI_PORT}/nifi-api/system-diagnostics | jq '.systemDiagnostics.aggregateSnapshot | {totalQueued: .totalQueued, totalNonLoopingFlowFiles: .totalNonLoopingFlowFiles, heapUsed: .heapUsed, heapUtilization: .heapUtilization}'

Example output:

{
  "totalQueued": 1205,
  "totalNonLoopingFlowFiles": "1205",
  "heapUsed": "350 MB",
  "heapUtilization": "70%"
}

Monitor these trends over time. A steadily increasing queue count or heap utilization near 90% indicates a problem.

Failure Modes and Recovery

Despite proactive monitoring, failures happen. This section outlines common failure modes, how to detect them, and recover.

Failure Mode 1: NiFi Node Out of Memory (OOM)

Symptoms:

• NiFi process crashes or becomes unresponsive
• Logs show java.lang.OutOfMemoryError: Java heap space
• Metrics show heap utilization near 100%

Detection:

• Prometheus alert on nifi_jvm_heap_utilization > 0.9
• Check via API: curl -k https://{NIFI_HOST}:{NIFI_PORT}/nifi-api/system-diagnostics | jq '.systemDiagnostics.aggregateSnapshot.heapUtilization'

Recovery:

1. Restart the NiFi node if it's down.
2. Increase heap size in bootstrap.conf (e.g., java.arg.2=-Xmx8g).
3. Restart NiFi.
4. Investigate flow design for memory leaks (e.g., large content in attributes, unbounded queues).

Failure Mode 2: Connection Backpressure

Symptoms:

• FlowFiles accumulate in a connection beyond its backpressure threshold
• Upstream processor stops running
• Data latency increases

Detection:

• NiFi UI shows queue full, backpressure engaged
• API: curl -k https://{NIFI_HOST}:{NIFI_PORT}/nifi-api/connections/{CONNECTION_ID} | jq '.status.aggregateSnapshot.percentUseCount'
• Alert on nifi_amount_flowfiles_queued / nifi_connection_backpressure_threshold > 1

Recovery:

1. Identify why downstream is slow: check processor logs, error queues, or external system (e.g., database, Kafka).
2. If transient, allow queue to drain.
3. If permanent, scale out downstream processors or increase backpressure threshold (careful with memory).

Failure Mode 3: Reporting Task Stopped Sending Metrics

Symptoms:

• No recent data in Grafana, Prometheus target shows down
• NiFi reporting task state is STOPPED or FAILED

Detection:

• Prometheus target health down alert
• Check task state: curl -k https://{NIFI_HOST}:{NIFI_PORT}/nifi-api/reporting-tasks/{TASK_ID} | jq '.status.runStatus'

Recovery:

1. Restart the reporting task via API or UI.
2. Check NiFi logs for errors in the reporting task.
3. If the task configuration is corrupted, recreate it.

Operations Checklist

Use this daily/weekly checklist to ensure comprehensive monitoring and quick response.

Daily Checks

• [ ] Verify Prometheus targets are all up for NiFi nodes.
• [ ] Check NiFi system diagnostics: curl -k https://{NIFI_HOST}:{NIFI_PORT}/nifi-api/system-diagnostics | jq '.systemDiagnostics.aggregateSnapshot | {heapUtilization, cpuUtilization, totalQueued}'
• [ ] Review alert manager for any active alerts.
• [ ] Look at Grafana dashboard for anomalies (queue buildup, error counts).

Weekly Checks

• [ ] Review NiFi logs for warnings and errors (grep -i 'ERROR\|WARN' {NIFI_HOME}/logs/nifi-app.log).
• [ ] Check disk space on NiFi nodes (df -h), ensure content repositories have enough free space (at least 20%).
• [ ] Verify backup of flow configuration: ls -l {NIFI_HOME}/conf/flow.xml.gz and test restore to a non-production environment.
• [ ] Review resource usage trends in Grafana; plan capacity if growth is steady.

Monthly Checks

• [ ] Update monitoring dashboards to reflect flow changes.
• [ ] Test alerting rules by simulating a failure (e.g., stop a processor and ensure alert fires).
• [ ] Review and update documentation for recovery procedures.
• [ ] Check for NiFi updates and security patches.

Conclusion

Effective NiFi monitoring and alerting requires a systematic approach: know your environment, configure metrics export safely, verify data flow, and prepare for common failures. Use the commands and examples in this guide as a starting point.

Start with one low-risk improvement: enable Prometheus reporting on a test NiFi instance, set up a Grafana dashboard with basic metrics, and create an alert for heap utilization. Record the baseline, run it for a week, and refine thresholds. Then expand to other failure modes and integrate with your incident response process.

Remember: observability is not just about collecting metrics; it's about using them to make informed decisions quickly. Keep your monitoring stack as reliable as your data flows.