NiFi Architecture Explained with Practical Examples

Intro

Apache NiFi is a powerful dataflow automation tool, but its architecture can feel overwhelming at first. This guide explores NiFi architecture with practical examples, helping operators move from an observed problem to a verified result. Whether you are a developer, DevOps consultant, or part of a technical startup team, you will learn how to connect NiFi components, design reliable data flows, and operate them safely.

We focus on operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths. Every recommendation is version-scoped, observable, and reversible where the technology permits.

Version and Environment Inventory

Before making any change to a NiFi deployment, you must know exactly what you are working with. Start by identifying the installed version and deployment topology. This prevents applying guidance meant for a different version or setup.

Read-only observation

To check the NiFi version from the command line, use the nifi.sh script with the status argument (on a standard installation):

/path/to/nifi/bin/nifi.sh status

Expected output on NiFi 1.x includes a line like:

Java home: /usr/lib/jvm/java-11-openjdk-amd64
NiFi home: /opt/nifi

Bootstrap File: /opt/nifi/conf/bootstrap.conf

2023-08-01 12:00:00,000 INFO [main] org.apache.nifi.bootstrap.Command - Apache NiFi is currently running, PID 12345

For NiFi 2.x, the script may have changed; always consult the official documentation for your version. For containerized deployments, use:

docker exec <container_name> /opt/nifi/bin/nifi.sh status

Prerequisites

Java version compatible with your NiFi version (NiFi 1.20+ requires Java 11 or 17; NiFi 2.0 requires Java 21).
Access to the NiFi installation directory or container.
Read permissions on configuration files if inspecting them.

Blast radius and change

If you determine the installed version is 1.20.0 and you intended to use features from 2.0, the smallest justified change is to upgrade to a supported version. Before doing so, back up your conf directory and flow definitions. The blast radius includes all processors and controller services because they may have compatibility changes.

Verification

After an upgrade, verify the version again with nifi.sh status or the REST API:

curl -k https://localhost:8443/nifi-api/system-diagnostics

Expected output includes "version":"2.0.0" in the JSON response. If you see a different version, roll back using your backup and consult migration notes.

Safe Configuration Path

Safe configuration in NiFi means separating observation from intervention. Many production incidents start with a well-intentioned change that was not scoped or reversible. Here is a practical path for modifying a processor property.

Example: Change a processor's scheduling period

Suppose you need to adjust the run schedule of a GetFile processor from 60 seconds to 10 seconds to reduce latency.

Observe current state via the NiFi REST API (replace <nifi-host> and <processor-id>):

curl -k -u admin:password https://<nifi-host>:8443/nifi-api/processors/<processor-id>

Look for the schedulingPeriod field in the JSON response. It may show "schedulingPeriod":"60 sec".

Verify prerequisites: You need the processor ID. In the NiFi UI, right-click the processor and select "View Configuration" to find the ID in the URL, or use the API to list processors in a process group.

Make the smallest change: Use a PUT request to update only the scheduling period. Construct a JSON payload with the current revision and the new value:

{
  "revision": { "clientId": "my-client", "version": 3 },
  "component": {
    "id": "<processor-id>",
    "config": {
      "schedulingPeriod": "10 sec"
    }
  }
}

Then submit:

curl -k -X PUT -u admin:password -H "Content-Type: application/json" -d @update.json https://<nifi-host>:8443/nifi-api/processors/<processor-id>

Blast radius: This change affects only the specified processor's execution frequency. It does not alter data content or flow relationships. However, a shorter period may increase system load.

Verify: Fetch the processor again and confirm "schedulingPeriod":"10 sec". Also monitor queue sizes to ensure the processor keeps up.

Recovery: If the change causes issues (e.g., excessive CPU), revert the scheduling period to the original value using the same API with the correct revision.

Security note: Never put real credentials in scripts or examples. Use environment variables or secure vaults. For the example above, replace admin:password with a reference to a secure credential.

Verification and Diagnostics

Verification is about proving that the system behaves as expected after a change or during routine operations. NiFi provides several diagnostic tools.

Using nifi.sh diagnostics

The diagnostics command generates a report of system and flow information. Run:

/path/to/nifi/bin/nifi.sh diagnostics <output-directory>

This creates a zip file containing logs, configuration, and thread dumps. Review the system-diagnostics.txt for memory and processor load:

Total Memory: 16.0 GB
Used Memory: 3.2 GB
Free Memory: 12.8 GB

Checking processor status via API

To verify that a processor is running and has processed some data, use:

curl -k -u admin:password https://<nifi-host>:8443/nifi-api/processors/<processor-id>/status

Look for fields like "runStatus":"Running" and aggregate counts of flow files processed. If runStatus is "Stopped" unexpectedly, that is a failure signal.

Troubleshooting example

Assume a PutDatabaseRecord processor is failing. Check the processor's bulletin:

curl -k -u admin:password https://<nifi-host>:8443/nifi-api/processors/<processor-id>/bulletins

If you see an error like "Cannot get connection for url jdbc:...", verify the database connection string and credentials in the DBCPConnectionPool controller service. Make one change at a time and restart the processor.

Verification checklist

Compare before and after metrics (queue sizes, flow file counts, error rates).
Confirm expected output files exist or target systems received data.
For data integrity, use checksums or row counts.

Failure Modes and Recovery

Understanding common failure modes helps you prepare recovery procedures. Here are three practical scenarios.

1. Out of disk space

Symptom: Processors fail with "No space left on device" in logs, or flow files accumulate in queues.

Diagnosis: Check disk usage:

df -h /path/to/nifi/repositories

If usage is near 100%, identify large content or flowfile repositories.

Recovery (choose one, smallest first):

Increase disk space or add a new content repository location in nifi.properties (nifi.content.repository.directory.*).
Clean up old provenance events via the UI (Global menu -> Data Provenance -> delete old events).
If safe, remove old log files.

Verification: After freeing space, run df -h again to confirm usage below 80%. Restart affected processors.

2. Processor stuck in running state without progress

Symptom: Processor shows "Running" but no flow files pass through, and bulletins show no errors.

Diagnosis: Suspect a thread deadlock or long-running task. Use nifi.sh diagnostics to obtain thread dumps. Look for threads blocked on the same lock.

Recovery: Stop the processor. If it does not stop, restart the NiFi instance. After restart, if the issue recurs, inspect the processor's configuration and dependencies (e.g., a controller service that is not enabled).

Verification: Enable the processor and watch the queue size; it should start draining.

3. Flow file stuck in a loop

Symptom: A processor such as RouteOnAttribute sends a file back to itself, causing infinite loops.

Diagnosis: Inspect the data flow connections and routing rules. A file's lineage shows repeated passes through the same processor.

Recovery: Modify the routing logic to route failures to a LogAttribute processor or a dead-letter queue instead of looping back. Test the change on a single flow file.

Verification: After the change, no new files enter the loop; existing stuck files are manually moved or removed.

General recovery principles

Always capture state before intervening: export the flow definition (right-click on canvas -> Download flow) as a backup.
Document the exact command or API call used for recovery.
Test recovery procedures in a staging environment when possible.

Operations Checklist

Use this checklist for routine NiFi operations to prevent incidents and ensure quick recovery.

Daily checks (automated via script)

Create a shell script that runs each morning and alerts on anomalies:

#!/bin/bash
# Check NiFi is running
if /path/to/nifi/bin/nifi.sh status | grep -q "Apache NiFi is currently running"; then
  echo "NiFi is running"
else
  echo "NiFi is not running - alert!"
fi

# Check disk usage (warning at 80%)
USAGE=$(df -h /opt/nifi | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$USAGE" -gt 80 ]; then
  echo "Disk usage is $USAGE% - alert!"
fi

Schedule with cron:

0 7 * * * /usr/local/bin/nifi_daily_check.sh

Processor and controller service health

List all processors with their run status:

curl -k -u admin:password https://<nifi-host>:8443/nifi-api/processors

Look for any processor with "runStatus":"Invalid" or "Stopped" that should be running.

Ensure all required controller services are enabled:

curl -k -u admin:password https://<nifi-host>:8443/nifi-api/controller-services

Filter for "state":"ENABLED" where expected.

Backup and restore

Before any configuration change, export the flow:

In the NiFi UI, go to the root process group, right-click, and select "Download flow definition" (JSON). Store it with a timestamp:

mv downloaded_flow.json flow_backup_$(date +%Y%m%d).json

To restore, import the JSON via the UI or API.

Version-specific considerations

NiFi 2.x introduced the nipyapi changes and new Registry integration. Always check your version's documentation. For example, the API endpoint for process-groups changed from /nifi-api/process-groups/{id} to include additional query parameters.

Security checklist

Use HTTPS with valid certificates; never disable certificate validation in production.
Use NiFi's built-in user management or integrate with LDAP/OpenID Connect.
Rotate secrets (keystore passwords, database credentials) periodically and update them in nifi.properties or secure variables.
Audit access via provenance events.

Conclusion

NiFi architecture explained with practical examples becomes useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure.

As a next step, choose one low-risk verification for your NiFi setup: record the current state, run a documented check (e.g., nifi.sh status or a processor status API call), compare the result with the expected signal, and review dependencies such as Kafka, HDFS, and Apache Spark if they are part of your flow.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Start small, observe, and iterate.