Apache Airflow Common Errors and Fixes: A Practical Troubleshooting Guide

Intro

Apache Airflow common errors and fixes with practical examples should help operators move from an observed problem to a verified result. Start by identifying the installed version, deployment topology, prerequisites, and the exact component being inspected.

This article focuses on Apache Airflow common errors for developers, DevOps consultants and technical startup teams. It connects Apache Airflow fixes, Apache Airflow error messages, Apache Airflow debugging and Apache Airflow troubleshooting to commands, expected output, failure signals, and recovery decisions that match the selected technology.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Throughout this guide, we break down troubleshooting into four phases that map directly to the sections below:

• Version and Environment Inventory: Know your deployment.
• Safe Configuration Path: Change one thing at a time with recovery in mind.
• Verification and Diagnostics: Use read-only commands and logs to pinpoint root cause.
• Failure Modes and Recovery: Handle the most common failure classes with concrete steps.

We also provide an Operations Checklist at the end to keep on hand during incidents.

Version and Environment Inventory

Before you can fix an Apache Airflow problem, you must know exactly what you are running. Blindly applying a fix intended for Airflow 2.6 to an Airflow 3.0 deployment can make things worse. The Version and Environment Inventory is a structured, read-only snapshot of your system. It documents:

• The exact Airflow version.
• The executor in use (for example, CeleryExecutor, KubernetesExecutor, LocalExecutor).
• The metadata database backend (PostgreSQL, MySQL, SQLite) and version.
• How DAGs are distributed and synced (git-sync, volume mounts, CI/CD pipeline).
• Installed provider packages and their versions (for example, apache-airflow-providers-amazon, apache-airflow-providers-google).
• Deployment method (Docker Compose, Kubernetes Helm chart, bare metal, managed service like MWAA or Cloud Composer).

Why Version Matters: Airflow 2 vs Airflow 3

Airflow 3 introduced significant architectural changes. The scheduler is split into a DAG processor and a scheduler, and a new API server handles the REST API. The old airflow webserver process is gone. Commands and health check endpoints differ. For example:

• In Airflow 2, airflow webserver starts the web UI.
• In Airflow 3, you run airflow api-server and access the UI through it.

If you see a ModuleNotFoundError or a KeyError like 'dag_processor' in logs, you might be running a command intended for the wrong major version.

Read-Only Health Checks

Always start with read-only commands that do not change state. On Airflow 2.7+ and all 3.x versions, use:

airflow version
# Example output: 2.10.3

airflow info
# Prints detailed environment: Python version, platform, plugins, providers.

Check the health of critical components without restarting anything.

For Airflow 3, the unified health endpoint is:

curl -s http://localhost:8080/api/v2/monitor/health

Expected output is a JSON object with each component's status. For a healthy system, you should see something like:

{
  "metadatabase": {"status": "healthy"},
  "scheduler": {"status": "healthy"},
  "triggerer": {"status": "healthy"},
  "dag_processor": {"status": "healthy"}
}

Do not treat an HTTP 200 as proof that everything is fine. The endpoint returns 200 even if one component reports "unhealthy". Instead, parse the JSON and look for non-healthy statuses.

For Airflow 2.x without the unified endpoint, check each piece:

airflow jobs check --job-type SchedulerJob --hostname $(hostname)
# Returns exit code 0 if scheduler heartbeat is fresh.

airflow db check
# Validates database connectivity and schema.

If you use CeleryExecutor, also verify broker connectivity:

celery -A airflow.executors.celery_executor inspect ping
# Expected: {'celery@worker1': {'ok': 'pong'}}

Upgrade Paths and Migrations

When planning an upgrade from Airflow 2.7+ to Airflow 3.x:

1. Review release notes: Each release has a list of breaking changes. Check the Airflow documentation and provider changelogs.
2. Back up the metadata database: This is non-negotiable. Use pg_dump for PostgreSQL, mysqldump for MySQL.
3. Test representative DAGs in a staging environment before touching production.
4. Use airflow db migrate after upgrading the code. This command applies schema changes. Do not just restart the services; a restart without migration will leave the database schema incompatible and cause scheduler crashes with InvalidRequestError or ProgrammingError.

Example migration command:

airflow db migrate
# Output ends with "Database migrating done!"

Do not substitute a restart for a migration or a verified recovery plan.

Safe Configuration Path

Configuration changes are a common source of Airflow errors. A missing secret, a typo in airflow.cfg, or an incompatible setting can bring down the entire pipeline. The Safe Configuration Path ensures that changes are deliberate, isolated, and reversible.

Locating Configuration

Airflow loads configuration from the first file found in:

1. AIRFLOW_CONFIG environment variable.
2. airflow.cfg in the current working directory.
3. airflow.cfg in $AIRFLOW_HOME.
4. Default values in the codebase.

To see the exact path and current settings, run:

airflow config list

This prints all settings in a format like setting = value. Note that sensitive values such as database passwords are masked.

Common Configuration Mistakes

Let's look at a few real-world configuration errors and how to fix them safely.

1. sqlalchemy.exc.OperationalError when connecting to the database

Symptom: Scheduler logs show OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused.

Root Cause: sql_alchemy_conn is misconfigured or the database is down.

Safe Fix:

• Verify the connection string. It should be in the format postgresql+psycopg2://user:password@host:port/database.
• Check that the host and port are reachable. Use nc -zv localhost 5432 (read-only).
• If using environment variables, ensure they are set in the environment where the scheduler runs.

Example airflow.cfg snippet:

[database]
sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@postgres/airflow

After correcting, validate with airflow db check.

2. Scheduler cannot read DAG files due to wrong dags_folder

Symptom: DAGs are not visible in the UI. Logs show FileNotFoundError or No such file or directory when trying to read dags_folder.

Root Cause: The dags_folder path is wrong or the user running Airflow lacks permissions.

Safe Fix:

• Confirm the absolute path. If using Docker, ensure the volume is mounted correctly.
• Check permissions: ls -ld /opt/airflow/dags should be accessible by the airflow user.
• Use airflow dags list-import-errors to see specific file-level errors.

Example command to list import errors:

airflow dags list-import-errors
# Output for no errors: No data found
# Otherwise, shows filename and traceback.

3. Airflow 2 vs Airflow 3 configuration differences

Airflow 3 moved many settings to a new [api_server] section. If you set web_server_port expecting it to work, it will be ignored. Instead, use:

[api_server]
port = 8080

Always consult the specific version's configuration reference before changing a setting.

Applying Configuration Changes Safely

• Never edit airflow.cfg directly on a running system without a plan. Use a version-controlled configuration, environment variables, or a secret manager.
• Change one setting at a time and record the change, the reason, and the expected effect.
• Restart only the affected component. In Airflow 2, if you change scheduler settings, restart only the scheduler, not the webserver and workers.
• Use airflow config get to verify that the new value is picked up before restarting:

airflow config get database sql_alchemy_conn
# Expected: postgresql+psycopg2://airflow:airflow@postgres/airflow

Verification and Diagnostics

After making a change, or when diagnosing an issue, you need to verify the state and collect diagnostics. This phase is about reading logs, checking task states, and using Airflow's built-in debugging tools.

Viewing Task Logs

Airflow stores logs for each task instance. The easiest way is via the UI, but on a server you can use the CLI:

airflow tasks logs example_dag task_1 2024-01-01

Output is the full log of that task instance. To follow logs in real time:

airflow tasks run example_dag task_1 2024-01-01 --local

This runs the task locally without scheduling, which is useful for debugging.

Diagnostic Commands

Here are key commands for diagnostics:

# Show task instances and their states for a DAG run
airflow tasks states-for-dag-run example_dag 2024-01-01T00:00:00+00:00

# List DAGs and their status
airflow dags list

# Show details of a specific DAG
airflow dags show example_dag

For scheduler health, check the scheduler logs. Look for lines like:

[2024-01-01 12:00:00,000] {scheduler_job.py:701} INFO - Starting the scheduler

If the scheduler is not processing DAGs, look for errors about parsing, such as:

ERROR - Dag import failed

The scheduler log will include the traceback, which often points to the exact line in your DAG file.

Using airflow tasks test for Dry Runs

Before deploying a change, test a task locally using airflow tasks test:

airflow tasks test my_dag my_task 2024-01-01

This executes the task in a single process without touching the scheduler or metadata database state for the run. It is a great way to catch Python errors, missing imports, or failing assertions.

Analyzing the Metadata Database Directly

Sometimes, you need to query the metadata database to understand task state. For example, to find all failed task instances in the last day:

SELECT dag_id, task_id, execution_date, state
FROM task_instance
WHERE state = 'failed'
  AND execution_date >= NOW() - INTERVAL '1 day';

In PostgreSQL, connect via psql -h postgres -U airflow. In MySQL, use mysql -h mysql -u airflow -p.

This can reveal patterns: a single task failing consistently might point to an external dependency issue, while many tasks across DAGs failing simultaneously suggests a scheduler or infrastructure problem.

Failure Modes and Recovery

Airflow failures fall into predictable categories. Here are the most common ones, with symptoms, root cause analysis, and recovery steps.

1. DAG Import Errors

Symptoms: DAG is not visible in UI or shows as broken. airflow dags list-import-errors shows errors.

Common Causes:

• Syntax error in the DAG file.
• Missing Python package (e.g., ImportError: No module named 'psycopg2').
• Permission issue reading the file.

Recovery Steps:

1. Run airflow dags list-import-errors to identify the exact file and error.
2. Fix the syntax or dependency.
3. For missing package, install it in the environment. For example, pip install apache-airflow-providers-postgres.
4. Restart the scheduler or DAG processor. In Airflow 2, the scheduler parses DAGs periodically, so the fix may be picked up automatically within min_file_process_interval (default 30 seconds). In Airflow 3, the DAG processor might need a restart.
5. Verify with airflow dags list and check that the DAG appears with no import errors.

2. Scheduling Delays or Missed Runs

Symptoms: DAG runs are late or never start. Scheduler logs show high CPU or long parsing times.

Common Causes:

• Too many DAGs or overly complex DAG parsing.
• Scheduler is overwhelmed due to high max_threads or insufficient resources.
• Time zone mismatch: DAG schedule is defined in a timezone that does not match scheduler timezone.

Recovery Steps:

1. Check scheduler health: airflow jobs check --job-type SchedulerJob --hostname $(hostname).
2. Review scheduler logs for slow parsing. Enable debug logging temporarily: AIRFLOW__LOGGING__LOGGING_LEVEL=DEBUG.
3. Optimize DAGs by moving heavy computations out of top-level code. For example, avoid making database connections at DAG definition time.
4. Increase scheduler resources or reduce scheduler.max_threads.
5. Verify schedule timezone: In the DAG, set schedule='0 6 *' with start_date=datetime(2024,1,1, tzinfo=pendulum.timezone('America/New_York')). Ensure the scheduler has the same timezone or adjust.

3. Task Failures and Retries

Symptoms: Task instance shows failed state in UI. Logs show an exception.

Recovery Steps:

1. Retrieve logs: airflow tasks logs my_dag my_task 2024-01-01.
2. Identify the root cause. Common errors:

• psycopg2.OperationalError: database unreachable.
• requests.exceptions.ConnectionError: external API down.
• ValueError from data validation.

1. Fix the underlying issue.
2. Clear the task instance to rerun:

airflow tasks clear my_dag --task-regex my_task --start-date 2024-01-01 --end-date 2024-01-01

1. Monitor the rerun to ensure success.

4. Zombie Tasks and Stuck Runs

Symptoms: Task instance remains in running state indefinitely, even after the worker crashed.

Root Cause: Worker failure without proper cleanup. Celery tasks can become zombies if the worker is killed.

Recovery Steps:

• In Airflow 2, the scheduler periodically detects and kills zombie tasks. You can adjust scheduler.task_queued_timeout and scheduler.task_failure_rate.
• Manually mark the task as failed if it cannot be killed:

UPDATE task_instance
SET state = 'failed',
    end_date = NOW(),
    duration = 0
WHERE dag_id = 'my_dag' AND task_id = 'my_task' AND state = 'running';

Be careful with direct database updates; always have a backup.

5. Metadata Database Issues

Symptoms: Scheduler crashes with OperationalError or InterfaceError when accessing the database.

Root Cause: Database is down, connection limit exceeded, or schema mismatch.

Recovery Steps:

1. Check database connectivity: airflow db check.
2. If using PostgreSQL, check connection count: SELECT count(*) FROM pg_stat_activity;. If at max, increase max_connections or reduce Airflow's sql_alchemy_pool_size.
3. For schema mismatch after upgrade, run airflow db migrate.

6. Secrets and Connection Errors

Symptoms: Tasks fail with KeyError or ConnectionNotDefined when trying to access connections.

Root Cause: The connection ID does not exist in the metadata database or environment variable not set.

Recovery Steps:

1. List connections: airflow connections list. Find the required conn_id.
2. If missing, add it via UI or CLI. To add a PostgreSQL connection:

airflow connections add 'postgres_default' \
    --conn-type 'postgres' \
    --conn-host 'postgres' \
    --conn-login 'airflow' \
    --conn-password 'airflow' \
    --conn-port 5432

1. If using environment variables for secrets, ensure they are set in the environment of the worker and scheduler. Test with echo $AIRFLOW_CONN_POSTGRES_DEFAULT.

Operations Checklist

Keep this checklist handy during incidents. It summarizes the safe sequence:

1. Record current state

• Run airflow version and airflow info.
• Note the time and any recent changes.

1. Check overall health

• Airflow 2: airflow jobs check --job-type SchedulerJob --hostname $(hostname) and airflow db check.
• Airflow 3: curl -s http://localhost:8080/api/v2/monitor/health and inspect each component's status.

1. Identify the failing component

• Is it the scheduler, webserver/api-server, worker, or database? Use logs and health checks.

1. Gather logs

• For task failures: airflow tasks logs <dag_id> <task_id> <execution_date>.
• For scheduler issues: view scheduler logs, typically in $AIRFLOW_HOME/logs/scheduler/latest/.

1. Check DAG import errors

• airflow dags list-import-errors.

1. Perform root cause analysis

• Look for common patterns: database connection, configuration typo, missing dependency, code bug.

1. Apply the smallest safe fix

• Change one variable at a time.
• Prefer environment variable override or a backup copy of the config file.

1. Verify the fix

• Re-run health checks.
• For DAG fix, test with airflow tasks test <dag_id> <task_id> <execution_date>.
• If task failed, clear and rerun: airflow tasks clear <dag_id> --task-regex <task_id> --start-date <date> --end-date <date>.

1. Document and communicate

• Note the incident timeline, root cause, and fix in your incident log.
• Update runbooks and alerting if necessary.

By following this checklist, you minimize downtime and avoid making changes that introduce new problems.

Conclusion

Apache Airflow common errors and fixes with practical examples is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure.

We have covered the essential steps: inventory your environment, make configuration changes safely, verify with diagnostics, and recover from common failures. Remember to always separate observation from intervention, protect credentials, and document recovery steps before you need them.

As a next step, choose one low-risk verification for Apache Airflow common errors, record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as Apache Spark, NiFi and Apache Hop if they are part of your pipelines.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.