Apache Airflow Upgrade and Migration: A Practical Implementation Guide

Intro

Apache Airflow upgrades and migrations are critical operations that require careful planning, precise execution, and thorough verification. Whether you are moving from Airflow 2 to Airflow 3, applying a minor version update, or migrating to a new infrastructure, the process demands a systematic approach to avoid downtime, data loss, or broken DAGs.

This guide is designed for developers, DevOps engineers, and technical startup teams responsible for maintaining Airflow environments. It provides practical, hands-on instructions for every phase: assessing your current setup, preparing for the upgrade, executing the migration, verifying success, and rolling back if necessary. We focus on real-world commands, expected outputs, and recovery strategies, ensuring you can operate with confidence.

Throughout this article, we emphasize operational safety: observe before changing, limit the blast radius, protect sensitive data, and always define recovery steps before you need them. By the end, you will have a clear, repeatable process for Apache Airflow upgrades and migrations, backed by concrete examples.

Version and Environment Inventory

Before any upgrade, you must understand your current Airflow environment in detail. Start by documenting the version, deployment topology, and key components. This inventory serves as the foundation for planning and rollback.

Step 1: Check Airflow Version and Components

Run the following command to determine the installed Airflow version:

airflow version

Expected output example:

2.9.3

If you are using Airflow 3, the output will start with 3.x.x. Note the major version because Airflow 3 introduces significant architectural changes, such as a separate API server and DAG processor.

Next, identify the executor, metadata database, and deployment method. Check your airflow.cfg or environment variables:

airflow config get-value core executor
airflow config get-value database sql_alchemy_conn

Expected outputs:

CeleryExecutor
postgresql+psycopg2://airflow:airflow@localhost/airflow

Common executors include SequentialExecutor, LocalExecutor, CeleryExecutor, and KubernetesExecutor. The metadata database might be PostgreSQL, MySQL, or SQLite. Deployment methods often include Docker, Kubernetes, or bare metal.

Also list installed providers:

airflow providers list

This shows provider packages and versions, which are crucial for compatibility checks.

Step 2: Health Checks (Read-Only)

Before making any changes, run read-only health checks to ensure the system is stable. In Airflow 2, use:

airflow jobs check --job-type SchedulerJob --hostname $(hostname)

Expected output (if healthy):

Found one alive job.

For database connectivity:

airflow db check

Expected output:

Connection successful.

In Airflow 3, the health endpoint provides detailed component status. Access the API:

curl -s http://<airflow-host>/api/v2/monitor/health

Example response:

{
  "metadatabase": {"status": "healthy"},
  "scheduler": {"status": "healthy"},
  "triggerer": {"status": "healthy"},
  "dag_processor": {"status": "healthy"}
}

Do not rely solely on a 200 status code; inspect each component's status.

Step 3: Document DAGs and Dependencies

Inventory your DAGs and note any critical ones for testing after upgrade:

airflow dags list

Also, record the location of DAG files and any custom plugins or hooks, as these may need updating.

Safe Configuration Path

When upgrading, configuration changes must be handled carefully to avoid breaking existing workflows. This section outlines a safe approach to validating and applying configuration updates.

1. Review Release Notes and Compatibility

Before upgrading, read the release notes for the target version. Pay attention to:

• Deprecated features
• Breaking changes
• Provider compatibility

For example, upgrading from Airflow 2.9 to 2.10 may deprecate certain operators. If you use providers like apache-airflow-providers-amazon, ensure they support the new version. Check provider compatibility using:

pip check

Or consult the provider's documentation.

2. Backup Configuration and Metadata Database

Always back up your airflow.cfg and any environment-specific settings:

cp $AIRFLOW_HOME/airflow.cfg $AIRFLOW_HOME/airflow.cfg.backup-$(date +%Y%m%d)

For the metadata database, use your DB's backup tool. For PostgreSQL:

pg_dump -U airflow airflow > airflow_db_backup_$(date +%Y%m%d).sql

For MySQL:

mysqldump -u airflow -p airflow > airflow_db_backup_$(date +%Y%m%d).sql

3. Test Changes in a Staging Environment

If possible, replicate your production environment and test the upgrade there first. Run your most critical DAGs in staging and verify they complete successfully.

4. Apply Configuration Changes Incrementally

After upgrading the Airflow version, you may need to adjust configurations. For example, in Airflow 3, the separation of API server and DAG processor requires new settings. Instead of making multiple changes at once, change one setting at a time and verify its effect.

For instance, if you need to update the executor to KubernetesExecutor, change it in airflow.cfg and restart the scheduler, then check logs for errors.

airflow config set core executor KubernetesExecutor

But note: changing the executor may require additional infrastructure setup.

5. Use airflow db migrate

After installing a new version, run the database migration command. This is mandatory for both minor and major upgrades.

airflow db migrate

Expected output (truncated):

INFO  [alembic.runtime.migration] Running upgrade 2.9.3 -> 2.10.0, add column to task_instance

Do not skip this step; a restart without migration can lead to schema mismatches and errors.

Verification and Diagnostics

After an upgrade, thorough verification is essential to ensure all components are functioning correctly. Use the following methods to diagnose any issues.

1. Verify Airflow Version and Components

Run airflow version again to confirm the new version.

Check that all expected processes are running. In Airflow 2, you can use:

ps aux | grep airflow

In Airflow 3, use the health endpoint as described earlier:

curl -s http://<airflow-host>/api/v2/monitor/health

2. Check Scheduler and Database Health

Re-run the health check commands:

airflow jobs check --job-type SchedulerJob --hostname $(hostname)
airflow db check

If the scheduler is not healthy, inspect its logs. Typical log location: $AIRFLOW_HOME/logs/scheduler/latest/.

3. Validate DAG Parsing

Ensure all DAGs are parsed without errors:

airflow dags list-import-errors

Expected output if no errors:

No data found

Or, for each DAG:

airflow dags list

Check that the DAGs appear with correct schedules.

4. Run a Test DAG

Trigger a simple test DAG or use airflow tasks test to execute a single task:

airflow tasks test example_bash_operator runme_0 2024-01-01

This command runs the task locally without affecting the scheduler. It helps verify task execution and dependencies.

For a more comprehensive test, trigger a DAG run via the UI or CLI:

airflow dags trigger -e 2024-01-01 example_bash_operator

Then monitor its state:

airflow dags list-runs -d example_bash_operator

5. Monitor Logs and Metrics

After triggering test runs, examine logs for errors:

tail -f $AIRFLOW_HOME/logs/dag_id=example_bash_operator/run_id=manual__2024-01-01T00:00:00/task_id=runme_0/attempt=1.log

If you use a monitoring system like Prometheus, check Airflow metrics for anomalies.

Failure Modes and Recovery

Despite careful planning, upgrades can fail. This section covers common failure scenarios and how to recover from them.

1. Database Migration Failure

If airflow db migrate fails, the error message usually indicates the cause. Common issues include missing extensions, permission problems, or incompatible database versions.

Recovery steps:

• Restore the database from backup.
• Fix the underlying issue (e.g., install required PostgreSQL extension).
• Re-run the migration.

Example rollback for PostgreSQL:

psql -U airflow airflow < airflow_db_backup_YYYYMMDD.sql

2. Scheduler Fails to Start After Upgrade

If the scheduler crashes after upgrade, check the logs for traceback. Common causes:

• Incompatible providers
• Configuration errors
• Missing dependencies

Troubleshoot:

airflow scheduler

Run in foreground to see errors directly. If a provider is incompatible, downgrade or upgrade it:

pip install apache-airflow-providers-amazon==<compatible-version>

If configuration error, revert the specific setting you changed.

3. DAG Import Errors

After upgrade, some DAGs may fail to parse due to deprecated operators. Use airflow dags list-import-errors to identify problematic DAGs. Then update the DAG code to use new operators or adjust import statements.

4. Performance Degradation

Sometimes upgrades introduce performance regressions. Monitor scheduler heartbeat, task throughput, and database load. If performance is unacceptable, consider rolling back to the previous version, but only after verifying that the database schema is compatible. Rolling back usually requires restoring the database backup because migration changes are often irreversible.

Rollback Plan

A solid rollback plan includes:

• Restoring the previous Airflow version (e.g., pip install apache-airflow==2.9.3)
• Restoring the database backup
• Restoring configuration files
• Restarting services

Example rollback commands:

pip install apache-airflow==2.9.3
psql -U airflow airflow < airflow_db_backup_YYYYMMDD.sql
cp $AIRFLOW_HOME/airflow.cfg.backup-YYYYMMDD $AIRFLOW_HOME/airflow.cfg
airflow db upgrade  # ensure schema matches version
airflow scheduler -D
airflow webserver -D

Always test the rollback procedure in staging before you need it in production.

Operations Checklist

Use this checklist before, during, and after an Airflow upgrade to ensure nothing is missed.

Pre-Upgrade

• [ ] Document current Airflow version, executor, database, providers, and deployment method.
• [ ] Review release notes and compatibility for target version.
• [ ] Back up airflow.cfg and any custom configurations.
• [ ] Back up metadata database.
• [ ] Identify critical DAGs for testing.
• [ ] Set up staging environment (if possible) and test upgrade there.
• [ ] Notify stakeholders and schedule maintenance window.

During Upgrade

• [ ] Install new Airflow version (e.g., pip install apache-airflow==X.Y.Z).
• [ ] Update provider packages if necessary.
• [ ] Run airflow db migrate and capture output.
• [ ] Restart services (scheduler, webserver, workers).
• [ ] Check health endpoints and logs.

Post-Upgrade Verification

• [ ] Run airflow version to confirm new version.
• [ ] Execute airflow db check and airflow jobs check.
• [ ] Verify DAG parsing (no import errors).
• [ ] Trigger test DAGs and monitor completion.
• [ ] Check logs for errors or warnings.
• [ ] Monitor performance metrics for a period (e.g., 24-48 hours).
• [ ] Update documentation with new version and any configuration changes.

Rollback Preparedness

• [ ] Keep backups easily accessible.
• [ ] Document exact rollback commands.
• [ ] Ensure team members know how to execute rollback.
• [ ] Test rollback in staging.

Conclusion

Apache Airflow upgrades and migrations require meticulous attention to detail, but with a structured approach, you can minimize risk and ensure a smooth transition. This guide has covered the essential steps: inventorying your environment, preparing a safe configuration path, verifying the upgrade, handling failures, and following a comprehensive checklist.

Remember that every Airflow environment is unique, so adapt these practices to your specific setup. Always prioritize observability, incremental changes, and recovery planning. By doing so, you will maintain a reliable Airflow platform that can evolve with your data workflows.

As a next step, choose one low-risk verification from this guide, such as running airflow jobs check, and incorporate it into your regular maintenance routine. Then, when the time comes for your next upgrade, you will be well-prepared to execute it confidently and recover quickly if needed.