Apache Airflow Troubleshooting: A Practical Field Guide with Commands, Logs, and Recovery

Introduction

When an Apache Airflow pipeline breaks at 2 a.m., the difference between a 15-minute fix and a 10-hour firefight comes down to one thing: having a structured approach to observation, diagnosis, and recovery. This guide provides practical, hands-on troubleshooting techniques for Apache Airflow, covering both the legacy 2.x series and the newer 3.x architecture.

We will focus on the most common failure points: the scheduler, workers, metadata database, DAG files, and the Airflow configuration. For each area, you will find read-only diagnostic commands, concrete error signatures, and minimal, reversible fixes. The goal is not just to solve the current problem, but to build a repeatable diagnostic workflow that reduces mean time to recovery (MTTR).

Airflow 3 introduced significant architectural changes, including a separate API server and DAG processor. Commands and health endpoints differ between versions, so this guide flags version-specific differences wherever they matter. We assume you have command-line access to the Airflow environment and sufficient permissions to run diagnostic commands and view logs.

Version and Environment Inventory

Before touching anything, establish a clear picture of the deployment. Knowing the exact version, executor, and topology prevents you from applying advice meant for a different setup.

Start with the version and component inventory:

airflow version

Example output:

3.1.0

If you are on Airflow 3, note that the airflow version output only tells you the core version. Providers have their own versioning. Check providers with:

airflow providers list

This command outputs a table of installed providers and their versions. Incompatible provider versions are a frequent source of cryptic errors, especially after an Airflow core upgrade.

Next, determine the executor and metadata database:

airflow config get-value core executor
airflow config get-value database sql_alchemy_conn

For Airflow 3, the database configuration may be under database rather than core. You can also inspect the full configuration with airflow config list, but be careful: that command can expose secrets. Use airflow config list --include-examples only in a safe environment, or redact sensitive values before sharing.

Document the deployment topology: Is this a single-node setup, a multi-node cluster with Celery or Kubernetes executor, or a managed service like Astronomer or MWAA? Write down how the scheduler, workers, web server, and DAG processor are deployed (bare metal, Docker, Kubernetes). This inventory will guide your troubleshooting steps.

For upgrades, especially from Airflow 2 to 3, review the official migration guide. Key breaking changes include the removal of the standalone DAG processor in the scheduler, the new API server, and changes to the configuration schema. Never attempt an upgrade without a metadata database backup:

airflow db backup

Airflow 2.7+ provides this command. For older versions, use your database's native backup tool (e.g., pg_dump for PostgreSQL). After backup, test the migration on a staging environment with representative DAGs before touching production.

Safe Configuration Path

Configuration errors are among the easiest to introduce and hardest to spot. A single typo in airflow.cfg or an environment variable can bring down the scheduler or cause tasks to silently misbehave. The safe configuration path is a discipline: observe, change one thing, verify, and know how to roll back.

Read-Only Configuration Inspection

Never edit configuration files blindly. First, see the current effective configuration:

airflow config list

This prints all settings in the active configuration, including any overrides from environment variables. To check a specific setting without dumping everything:

airflow config get-value core dags_folder

For Airflow 3, some sections have been renamed. For example, the DAG processor settings moved from scheduler to dag_processor. If get-value complains about a missing section, consult the current configuration reference for your version.

Minimal, Justified Changes

When you need to change a setting, make the smallest possible change and document it. For example, if the DAG processor is not picking up new DAGs, you might need to adjust the dag_dir_list_interval. Instead of guessing, check the current value:

airflow config get-value scheduler dag_dir_list_interval

Then set a new value only if justified. In Airflow 3, this setting moved to dag_processor:

airflow config get-value dag_processor dag_dir_list_interval

If you must change it, use an environment variable scoped to the affected component rather than editing the global airflow.cfg. For example:

export AIRFLOW__DAG_PROCESSOR__DAG_DIR_LIST_INTERVAL=60

This limits the blast radius and makes the change visible in the environment.

Secrets Management

Never put secrets in airflow.cfg or in DAG code. Airflow supports secret backends and environment variables. If you see a connection string with embedded credentials in the configuration, replace it with a reference to a secret backend. For example, in airflow.cfg:

[database]
sql_alchemy_conn = postgresql+psycopg2://airflow:${AIRFLOW_DB_PASSWORD}@postgres/airflow

Then set AIRFLOW_DB_PASSWORD in the environment. Airflow will expand the variable at runtime. For connections, use the AIRFLOW_CONN_ prefix, e.g., AIRFLOW_CONN_MY_DB=postgres://user:pass@host:5432/db. This avoids hardcoding secrets.

Verify After Change

After any configuration change, restart only the affected component and verify it is healthy before declaring victory. For example, if you changed a scheduler setting, restart the scheduler:

# In a systemd setup
sudo systemctl restart airflow-scheduler

Then check:

airflow jobs check --job-type SchedulerJob --hostname $(hostname)

In Airflow 3, you can also query the API server health endpoint. More on that in the next section.

Verification and Diagnostics

Effective troubleshooting relies on reliable health checks and log analysis. Airflow exposes several endpoints and commands to assess component health without changing anything.

Health Checks for Airflow 3

In Airflow 3, the API server provides a detailed health endpoint:

curl -s http://localhost:8080/api/v2/monitor/health

Example response (truncated for clarity):

{
  "metadatabase": {"status": "healthy"},
  "scheduler": {"status": "healthy", "latest_scheduler_heartbeat": "2025-01-15T10:30:00Z"},
  "triggerer": {"status": "healthy"},
  "dag_processor": {"status": "healthy"}
}

Do not rely on HTTP 200 alone. The response body contains the status of each subcomponent. If metadatabase is unhealthy, the API server may still return 200 because the API server process is running.

For Airflow 2, the equivalent health endpoint is /health, which returns a simple JSON with metadatabase and scheduler status. However, the Airflow 3 endpoint is more comprehensive.

Command-Line Health Checks

Airflow provides targeted commands to check specific components:

airflow jobs check --job-type SchedulerJob --hostname $(hostname)

This command exits with 0 if the scheduler has sent a heartbeat recently. You can also check the triggerer:

airflow jobs check --job-type TriggererJob --hostname $(hostname)

For metadata database connectivity:

airflow db check

This performs a simple query against the database and reports success or failure. If it fails, check the database connection string, network access, and credentials.

Log Analysis Fundamentals

Airflow logs are your primary diagnostic resource. The scheduler logs are usually in $AIRFLOW_HOME/logs/scheduler/latest/. The web server logs are in $AIRFLOW_HOME/logs/webserver/. For Docker deployments, use docker logs <container_name>.

Task instance logs are stored in $AIRFLOW_HOME/logs/dag_id=<dag_id>/run_id=<run_id>/task_id=<task_id>/attempt=<attempt>.log. You can also view them from the UI. When a task fails, first look at the task log for Python tracebacks or application errors. If the log is empty or missing, the task may have failed before logging started, which indicates an infrastructure issue (e.g., worker crash, OOM kill).

Use grep to filter large logs:

grep -i "error" $AIRFLOW_HOME/logs/scheduler/latest/*.log | tail -50

For remote logging (S3, GCS), ensure the log configuration is correct and credentials are valid. Remote log failures often appear as Could not read remote logs in the UI.

DAG Parsing Issues

A DAG that does not appear in the UI or does not schedule is often a parsing problem. Run the DAG parser manually:

python -c "from airflow.models import DagBag; d = DagBag(); print(d.import_errors)"

If there are import errors, they will be printed. Fix the code and re-run until clean. In Airflow 3, the DAG processor runs separately; you can also check its logs for parse errors.

Failure Modes and Recovery

Now let's walk through specific failure scenarios and how to recover from them.

Scheduler Not Scheduling

Symptoms: DAGs appear in the UI but tasks are not being scheduled; scheduler logs show errors; airflow jobs check --job-type SchedulerJob fails.

Diagnosis:

1. Check scheduler heartbeat:

   airflow jobs check --job-type SchedulerJob --hostname $(hostname)

1. Check scheduler logs for exceptions. Common causes: database connection pool exhaustion, long-running queries, or a stuck DAG processor.
2. Verify the metadata database is responsive:

   airflow db check

1. In Airflow 3, ensure the DAG processor is running and healthy. The scheduler no longer processes DAGs; it delegates to a separate DAG processor service.

Recovery:

• Restart the scheduler (and DAG processor in Airflow 3).
• If database pool exhaustion is suspected, increase sql_alchemy_pool_size in [database] section, but first check database connection limits.
• If a specific DAG is causing the scheduler to hang, temporarily pause or delete the DAG file and restart the scheduler to isolate the problem.

Worker OOM Kills

Symptoms: Tasks fail with exit code 137 or 143 (SIGKILL/SIGTERM); logs abruptly end; worker pod restart in Kubernetes.

Diagnosis:

• Check worker resource usage: free -m, top, or Kubernetes pod metrics.
• Look at kernel logs for OOM killer messages: dmesg | grep -i oom (requires root).
• Check task memory settings: airflow config get-value celery worker_concurrency or Airflow's worker_container_memory_request for Kubernetes executor.

Recovery:

• Reduce task parallelism or increase worker memory.
• Optimize the DAG to use less memory (e.g., avoid loading large data into memory).
• For Python tasks, consider using @task with execution_timeout and retries to limit impact.
• In Kubernetes, adjust resource limits in the pod template.

Metadata Database Connection Issues

Symptoms: Scheduler and web server fail to start; logs show OperationalError: could not connect to server; airflow db check fails.

Diagnosis:

• Test database connectivity using the same connection string:

  psql "$AIRFLOW__DATABASE__SQL_ALCHEMY_CONN" -c "SELECT 1"

(Adjust for your database client.)

• Check network reachability: telnet <db_host> 5432 or nc -zv <db_host> 5432.
• Check database server logs for connection rejections.

Recovery:

• If the database is down, restore it from backup or fix the underlying issue.
• If credentials are incorrect, update the secret in the secret backend or environment variable.
• If the connection string is malformed, correct it and restart affected components.

DAG Import Errors

Symptoms: DAG not visible in UI; DagBag import errors; scheduler logs show ImportError or ModuleNotFoundError.

Diagnosis:

• Run python -c "from airflow.models import DagBag; d = DagBag(); print(d.import_errors)" to see the error.
• If the error is a missing module, install the required package in the scheduler/worker environment.
• If the error is a syntax error, fix the DAG file.

Recovery:

• Fix the code, re-upload or edit the DAG file, and wait for the next DAG parsing cycle (or trigger manually: airflow dags reserialize in Airflow 2, or restart the DAG processor in Airflow 3).
• Ensure the DAG file is in the correct dags_folder and has proper permissions.

Upgrading Database Schema

After an Airflow upgrade, the metadata database may need migration. Symptoms: Scheduler or web server reports version mismatch; logs show Your database schema is not up to date.

Diagnosis:

• Run airflow db check-migration (Airflow 2.7+) or airflow db migrate --check to see if migration is needed.

Recovery:

• Backup the database.
• Run airflow db migrate to apply schema changes.
• Restart all components.
• Verify with airflow db check and health endpoints.

Common Pitfalls and How to Avoid Them

Even experienced operators fall into recurring traps. Here are the most common pitfalls and how to avoid or recover from them.

Pitfall 1: Restarting as a Panacea

Why it happens: Restarting the scheduler or web server is quick and often appears to fix transient issues.

How to avoid: Treat restart as a last resort after diagnosis. A restart may mask a deeper problem that will recur. Instead, examine logs and health checks before restarting. If you do restart, do it component by component and verify health afterward.

Recovery: If you restarted and the problem persists or worsens, immediately gather logs from before the restart (if available) and perform a systematic diagnosis.

Pitfall 2: Ignoring Version Differences

Why it happens: Many tutorials and answers online are for Airflow 2, but users may be on Airflow 3. Commands and configuration keys have changed.

How to avoid: Always check airflow version and consult the official documentation for your specific version. Use airflow config get-value to verify setting names before editing.

Recovery: If you applied a change based on a wrong version, revert it to the previous value and consult the correct docs.

Pitfall 3: Hardcoding Secrets in DAGs or Configs

Why it happens: It is convenient during development, and developers may forget to remove them before production.

How to avoid: Use Airflow Connections and Variables managed via the UI or secret backends. Enforce code reviews and static checks for secrets. Use environment variables for configuration.

Recovery: If a secret leaked, rotate it immediately, remove it from the DAG/config, and consider using a secret backend. Audit logs to check for unauthorized access.

Pitfall 4: Neglecting Database Backups Before Migrations

Why it happens: Migrations are often seen as routine, and backups take time and storage.

How to avoid: Make backups a mandatory step in any upgrade or migration procedure. Automate backups and test restore procedures regularly.

Recovery: If a migration fails, restore from backup and analyze the failure in a staging environment before retrying.

Pitfall 5: Not Monitoring After Recovery

Why it happens: Once the incident is resolved, teams move on to other work.

How to avoid: Set up monitoring and alerting for scheduler heartbeats, database health, and task failure rates. Review logs periodically for recurring warnings. Establish a post-incident review to update runbooks.

Recovery: If the same issue recurs, the runbook needs updating. Capture the new knowledge and share it with the team.

Operations Checklist

The following checklist consolidates the essential steps for troubleshooting Apache Airflow. Use it as a quick reference during incidents.

Ownership note: The on-call engineer is responsible for initial diagnosis and safe intervention. The tech lead must approve any configuration change that affects multiple components. Post-incident reviews are owned by the team lead and should occur within 24 hours, with a follow-up review weekly to track action items.

Conclusion

Apache Airflow is a powerful orchestrator, but its distributed nature means failures can be complex. By following a structured approach—starting with environment inventory, using read-only health checks, making minimal changes, and verifying recovery—you can reduce downtime and prevent recurrences.

This guide has covered version-specific diagnostics, safe configuration practices, common failure modes, and pitfalls. The key takeaways are:

• Always check the version and component health before intervening.
• Use read-only commands to observe, then make one small change at a time.
• Protect secrets and back up the database before migrations.
• Treat logs as your primary evidence, and learn to read them efficiently.
• Document every incident and iterate on your runbooks.

As a next step, choose a low-risk verification on your current Airflow deployment: run airflow version, check the health endpoint, and inspect scheduler logs for any silent errors. This baseline will prepare you for the next incident. For deeper integration, review how Apache Airflow interacts with related data tools like Apache Spark, NiFi, or Apache Hop, but always scope any changes to the Airflow component at hand.

A reliable troubleshooting workflow turns chaos into a series of checkable steps, reduces mean time to recovery, and builds confidence in your data pipelines.