Apache Airflow Advanced Concepts: Version-Scoped Operations Guide with Practical Examples

Intro

Apache Airflow is a powerful workflow orchestration platform, but its advanced concepts often remain opaque to operators who only know how to write DAGs. This guide explains advanced concepts through practical examples: version and environment inventory, safe configuration paths, verification and diagnostics, failure modes, and recovery. We focus on operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

The article is written for developers, DevOps consultants, and technical startup teams who need to move from an observed problem to a verified result. It connects Apache Airflow internals, architecture, deep-dive topics, and examples to commands, expected output, failure signals, and recovery decisions. We cover both Airflow 2 and Airflow 3, highlighting key differences such as the separated DAG processor and API server in Airflow 3.

By the end, you will have a set of concrete procedures for inspecting, configuring, and recovering Airflow deployments, with every recommendation tied to a version scope and a verification signal.

Version and Environment Inventory

Before any intervention, you must know exactly what you are running. This section provides a read-only inventory process to capture the relevant component, version range, prerequisites, and current state without making changes.

Identify Your Airflow Version and Topology

Run the following command to get the Airflow version:

airflow version

Expected output example:

2.10.3

For Airflow 3, the output might be 3.0.0 or later. The version number determines which commands and health check endpoints are available. Always document the version before troubleshooting.

Next, determine the executor and metadata database. You can inspect the configuration without changing it:

airflow config get-value core executor
airflow config get-value database sql_alchemy_conn

Example output:

CeleryExecutor
postgresql+psycopg2://airflow:airflow@postgres/airflow

Note: In production, avoid printing the full SQLAlchemy connection string if it contains a password. Use airflow config get-value database sql_alchemy_conn only in a secure environment, or redact the password section if logging. Better, use environment variables and never hardcode secrets in airflow.cfg.

For deployment method, check if you are using Docker, Kubernetes, or a standalone setup:

# Docker
docker ps --filter "name=airflow"

# Kubernetes
kubectl get pods -n airflow

If on Kubernetes, also check the Helm chart version:

helm list -n airflow

Record these outputs along with a timestamp. For example:

date -u +"%Y-%m-%dT%H:%M:%SZ" && airflow version

Airflow 2 vs Airflow 3: Component Differences

Airflow 3 introduces a separate DAG processor component and an API server. In Airflow 2, the scheduler often parses DAGs in-process. In Airflow 3, the DAG processor runs as its own process, and the API server handles REST requests. This affects health checks and resource allocation.

To see running processes, use:

ps aux | grep -E "airflow (scheduler|dag-processor|api-server|webserver)"

Expect output lines for each component. In Airflow 3, you should see separate processes for the DAG processor and API server. If you only see a scheduler and webserver, you are likely on Airflow 2.

Read-Only Health Checks Before Intervening

Health checks must be read-only and version-aware. In Airflow 2, the airflow jobs check command works for scheduler checks. In Airflow 3, use the API server's health endpoint.

For Airflow 2, check scheduler health:

airflow jobs check --job-type SchedulerJob --hostname $(hostname)

Expected output on success:

Found one alive job.

For Airflow 3, query the API server health endpoint:

curl -s http://localhost:8080/api/v2/monitor/health

Example response:

{
  "metadatabase": {
    "status": "healthy"
  },
  "scheduler": {
    "status": "healthy",
    "latest_scheduler_heartbeat": "2024-01-01T00:00:00Z"
  },
  "triggerer": {
    "status": "healthy",
    "latest_triggerer_heartbeat": "2024-01-01T00:00:00Z"
  },
  "dag_processor": {
    "status": "healthy",
    "latest_heartbeat": "2024-01-01T00:00:00Z"
  }
}

Do not treat HTTP 200 as proof that every component is healthy. Inspect each field. For example, if scheduler.status is "unhealthy" but HTTP status is 200, the overall system may still be degraded.

Check the metadata database connectivity with:

airflow db check

Expected output:

Connection successful.

If the command fails, you will see an error like sqlalchemy.exc.OperationalError. Fix the database connection before proceeding.

Upgrade Considerations (Airflow 2.7+ and 3.x)

When upgrading, never assume a restart is enough. Follow these steps:

1. Review release notes for the target version.
2. Check provider package compatibility. Use pip list | grep apache-airflow-providers to list installed providers.
3. Back up the metadata database. For PostgreSQL:

pg_dump -U airflow airflow > airflow_backup_$(date +%Y%m%d).sql

1. Test representative DAGs in a staging environment before production upgrade.
2. Run the database migration:

airflow db migrate

Do not run airflow db upgrade or restart services before migration completes. The db migrate command applies schema changes and may take time depending on database size.

Safe Configuration Path

Configuration changes are a common source of outages. This section explains how to observe the current configuration, make minimal changes, and verify the result.

Read-Only Configuration Inspection

Before changing any configuration, inspect the currently effective values. Use:

airflow config list

This outputs all configuration options with their current values, but be careful: it may include secrets. Instead, retrieve specific settings:

airflow config get-value core dags_folder
airflow config get-value scheduler scheduler_heartbeat_sec

Example output:

/opt/airflow/dags
5

For environment variables that override config, check with:

env | grep AIRFLOW__

Expected output includes lines like:

AIRFLOW__CORE__EXECUTOR=CeleryExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow

Again, redact secrets in logs. Use a tool like sed to mask passwords:

env | grep AIRFLOW__ | sed 's/://[^:]*:[^@]*@/://***:***@/'

Minimal Change Example: Increase Scheduler Heartbeat

Suppose you observe that the scheduler is missing heartbeats, causing DAG runs to be delayed. In Airflow 2, the default scheduler_heartbeat_sec is 5 seconds. You might want to increase it to reduce database load. However, increasing it too much can delay detection of scheduler failure.

To make a minimal change, first back up the current config file:

cp /opt/airflow/airflow.cfg /opt/airflow/airflow.cfg.bak.$(date +%Y%m%d)

Then edit airflow.cfg under [scheduler]:

[scheduler]
scheduler_heartbeat_sec = 10

Alternatively, set an environment variable before starting the scheduler:

export AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC=10

After restarting the scheduler, verify the change took effect:

airflow config get-value scheduler scheduler_heartbeat_sec

Expected output: 10.

Then check scheduler health again with airflow jobs check --job-type SchedulerJob --hostname $(hostname) or the health API in Airflow 3.

Configuration Secrets Management

Never store secrets directly in airflow.cfg. Use environment variables or a secrets backend. For example, set the database password via environment variable:

export AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:${DB_PASSWORD}@postgres/airflow

In production, use a secrets manager like HashiCorp Vault or AWS Secrets Manager. Airflow supports secrets backends via airflow.secrets configuration.

To test if secrets are resolvable, run a simple DAG that prints a connection URI without revealing the password. Better, use airflow connections get with a redacted output. For example:

airflow connections get my_postgres_conn

If the password is displayed, ensure your secrets backend is masking it.

Verification and Diagnostics

Verification is about confirming that the system is behaving as expected. This section provides commands and expected outputs for common diagnostic scenarios.

DAG Parsing and Scheduling Verification

Airflow parses DAG files to detect errors and update the DAG bag. In Airflow 2, the scheduler parses DAGs; in Airflow 3, the DAG processor does. To check parsing errors without triggering runs:

airflow dags list

Expected output is a table of DAG IDs, owners, and schedule intervals. If a DAG has errors, it may not appear or may show an error count.

To see details for a specific DAG:

airflow dags show example_dag

This outputs the DAG structure. If the DAG cannot be parsed, you will see an error.

In Airflow 3, you can query the API server to see DAG processing status:

curl -s http://localhost:8080/api/v2/dags?limit=100 | jq '.dags[] | {dag_id, is_paused, fileloc}'

This returns JSON with each DAG's ID, paused status, and file location.

Scheduler Health Deep Dive

The scheduler is critical. In Airflow 2, use airflow jobs check as shown earlier. Additionally, inspect the scheduler logs:

tail -n 100 /opt/airflow/logs/scheduler/latest/scheduler.log

Look for errors like Scheduler heartbeat timed out or Unable to acquire lock on database.

In Airflow 3, the scheduler and DAG processor have separate logs. Check DAG processor logs:

tail -n 100 /opt/airflow/logs/dag_processor/latest/dag_processor.log

Database Connectivity and Performance

Use airflow db check to verify connectivity. For performance issues, query the database directly. For example, to see the number of queued tasks:

psql -U airflow -d airflow -c "SELECT state, count(*) FROM task_instance GROUP BY state;"

Expected output example:

   state   | count
-----------+-------
 queued    |    15
 running   |     3
 success   |  1200
 failed    |     2

If the queued count grows over time, the scheduler may not be processing tasks fast enough. Check worker capacity or scheduler configuration.

API Server Health in Airflow 3

Airflow 3's API server is a separate component. Check its health:

curl -s http://localhost:8080/api/v2/monitor/health

As shown earlier, the response includes component statuses. If the API server is down, the health check will fail with a connection error. Ensure the API server process is running:

ps aux | grep "airflow api-server"

Failure Modes and Recovery

Understanding common failure modes and how to recover is essential for keeping Airflow reliable. This section covers specific failures, their symptoms, and recovery steps.

Scheduler Down or Heartbeat Lost

Symptoms: DAG runs are not triggered, tasks remain in queued state, and airflow jobs check reports no alive jobs.

Recovery steps:

1. Check if the scheduler process is running.
2. If not, start it: airflow scheduler (in Airflow 2) or airflow scheduler and airflow dag-processor (in Airflow 3).
3. If it is running but not heartbeating, inspect logs for database connection errors. Fix the database connection.
4. Verify with airflow jobs check or the health API.

Metadata Database Outage

Symptoms: airflow db check fails, scheduler and webserver log database connection errors.

Recovery steps:

1. Check database server status: pg_isready -h postgres -p 5432 for PostgreSQL.
2. If down, restart the database service.
3. Verify Airflow can connect: airflow db check.
4. If database is up but Airflow still cannot connect, check network, credentials, and firewall.
5. Once restored, ensure scheduler resumes heartbeat.

DAG Processor Stuck (Airflow 3)

Symptoms: New DAGs are not appearing, changes to DAG files are not reflected, and health endpoint shows dag_processor.status: unhealthy.

Recovery steps:

1. Check DAG processor logs for errors.
2. Restart the DAG processor: airflow dag-processor.
3. Verify health: curl -s http://localhost:8080/api/v2/monitor/health | jq '.dag_processor.status' should return "healthy".

Task Failures and Retries

Symptoms: Tasks fail with specific errors. Recovery depends on the error. Always check task logs first:

airflow tasks logs example_dag example_task 2024-01-01T00:00:00+00:00

If the error is transient (e.g., network timeout), retry the task:

airflow tasks retry example_dag example_task 2024-01-01T00:00:00+00:00

For a permanent fix, update the DAG code and re-run.

Upgrade Failure Recovery

If an upgrade fails, do not attempt to restart with the new version. Roll back to the previous version and restore the database backup.

Steps:

1. Stop Airflow services.
2. Restore the database from backup: psql -U airflow -d airflow < airflow_backup_20240101.sql
3. Reinstall the previous Airflow version: pip install apache-airflow==2.10.3
4. Start services and verify with health checks.

Always have a tested rollback plan before upgrading.

Operations Checklist

Use this checklist for any Airflow operation, whether it is a configuration change, upgrade, or troubleshooting session.

Before Any Change

• [ ] Record the current Airflow version: run airflow version and note the output.
• [ ] Identify the deployment topology: Docker, Kubernetes, or standalone. Check with docker ps or kubectl get pods -n airflow.
• [ ] Note the executor and metadata database: airflow config get-value core executor and airflow config get-value database sql_alchemy_conn (redact password).
• [ ] Capture current health state: run airflow jobs check (Airflow 2) or curl http://localhost:8080/api/v2/monitor/health (Airflow 3). Save the output.
• [ ] Back up the metadata database: pg_dump -U airflow airflow > airflow_backup_$(date +%Y%m%d).sql
• [ ] Review recent logs for errors: tail -n 200 /opt/airflow/logs/scheduler/latest/scheduler.log
• [ ] Ensure you have a rollback plan.

Making the Change

• [ ] Change only one setting at a time.
• [ ] Use environment variables or a secrets backend for sensitive values; never hardcode secrets.
• [ ] If editing airflow.cfg, first create a backup: cp /opt/airflow/airflow.cfg /opt/airflow/airflow.cfg.bak.$(date +%Y%m%d)
• [ ] Apply the change to a test environment first, if possible.

After the Change

• [ ] Verify the change took effect: airflow config get-value <section> <key> should return the new value.
• [ ] Check health again: airflow jobs check or health API endpoint.
• [ ] Monitor logs for new errors for at least a few minutes.
• [ ] Confirm DAG parsing is successful: airflow dags list should not show errors.
• [ ] If the change involves a database migration, run airflow db migrate before starting services.
• [ ] Document the change and the verification results, including timestamps.

Recovery Verification

• [ ] If a rollback was performed, verify the system is back to the previous stable state by checking version, config, and health.
• [ ] Confirm that no data was lost: check recent task instances in the database.

Conclusion

Apache Airflow advanced concepts become manageable when you apply version-scoped, observable, and reversible procedures. Every recommendation in this guide includes a command or signal to verify the outcome. Start by identifying your Airflow version and topology, then use read-only health checks before making any change. Limit changes to one scoped item, protect secrets, and always have a recovery plan.

As a next step, choose one low-risk verification from this article, such as checking the health endpoint in Airflow 3 or running airflow db check in Airflow 2. Record the current state, run the documented check, and compare the result with the expected signal. Review dependencies such as provider packages and database connectivity.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With these practices, you can keep your Airflow deployment healthy and resilient.