Apache Airflow Backup and Restore: Practical Examples for Reliable Recovery

Intro

Backing up Apache Airflow is not just about copying files. A reliable backup lets you restore the metadata database, DAG files, logs, and configuration so that your workflows can run again after an incident. This article gives you practical commands and checks for Airflow 2 and 3 so that you can back up and restore with confidence.

We focus on the steps that matter for operators: taking a consistent backup of the metadata database, preserving your DAGs and plugins, capturing configuration, and validating that a restore actually works. Along the way, we cover common failure modes and how to recover from them. The goal is to make your backup routine testable, repeatable, and safe.

Throughout, we use concrete examples with command output and expected results. We avoid placeholders and show real paths, environment variables, and validation checks.

Version and Environment Inventory

Before you back up anything, know exactly what you are running. Different Airflow versions change where data lives and how you validate a restore.

Start by checking the Airflow version and configuration:

airflow version

Expected output looks like:

3.0.2

or for Airflow 2:

2.10.4

Next, inspect the current configuration without exposing secrets. The airflow config list command shows effective settings but may include connection strings. For a safer read-only view, use:

airflow config get-value core sql_alchemy_conn

Replace sql_alchemy_conn with any key you need. This prints only the value, so be careful not to expose it in logs. For example, if your metadata database is PostgreSQL, you might see:

postgresql+psycopg2://airflow:****@postgres:5432/airflow

In many default setups, the password is masked, but verify before running this on a shared terminal.

Document the following details in your inventory:

• Airflow version (2.x or 3.x). In Airflow 3, the architecture introduced an API server and a separate DAG processor. This changes what you need to back up.
• Executor type (LocalExecutor, CeleryExecutor, KubernetesExecutor, etc.). Check with airflow config get-value core executor.
• Metadata database type and connection details. On Airflow 2, use airflow db check; on Airflow 3, use airflow db check as well if supported, or inspect the configuration.
• Deployment method: Docker, Kubernetes, bare metal, or managed service. This affects where the DAG files and logs are stored.
• DAG distribution: Are DAGs in a shared volume, Git repository, or object storage?

Health checks: For Airflow 3, the API server exposes /api/v2/monitor/health. Do not treat HTTP 200 as proof that all components are healthy. Instead, inspect the JSON body:

curl -s http://airflow-api:8080/api/v2/monitor/health | jq .

Expected output includes fields like:

{
  "metadatabase": {
    "status": "healthy"
  },
  "scheduler": {
    "status": "healthy"
  },
  "triggerer": {
    "status": "healthy"
  }
}

If any component reports unhealthy, investigate before backing up, because you may capture an inconsistent state.

For scheduler checks, use:

airflow jobs check --job-type SchedulerJob --hostname $(hostname)

Expected output:

Found one alive job.

If it reports no alive job, the scheduler is down and your backup may miss scheduling state.

Safe Configuration Path

Before making any changes, capture the current configuration so you can roll back if needed. The airflow.cfg file or environment variables define core behavior. Back them up explicitly.

Example: if you run Airflow in Docker with a config file mounted at /opt/airflow/airflow.cfg, copy it:

cp /opt/airflow/airflow.cfg /backup/airflow/airflow.cfg.$(date +%Y%m%d)

Verify the copy:

ls -l /backup/airflow/

You should see a timestamped file.

For Airflow 3, configuration may be managed via the API or environment variables. Use airflow config list to dump effective configuration and save it to a file, but be cautious with secrets. For example:

airflow config list > /backup/airflow/config-$(date +%Y%m%d).txt

Review the file and redact sensitive values before storing it in a shared location.

If you need to change a configuration value, do it in a controlled way. For example, to change the DAG processor poll interval on Airflow 3, edit airflow.cfg or set an environment variable, then restart only the DAG processor service. Verify the change took effect:

airflow config get-value dag_processor processor_poll_interval

Expected output:

30

If you see the old value, the service did not pick up the change. Check the restart and configuration loading.

When upgrading Airflow, never skip the metadata database backup. Also, test representative DAGs in a staging environment before upgrading production. For Airflow 2.7+ and 3.x, the recommended upgrade path includes:

1. Stop the scheduler and workers (for Airflow 2) or the corresponding components.
2. Back up the metadata database.
3. Review release notes and provider compatibility.
4. Run airflow db migrate.
5. Restart services and run health checks.

Do not substitute a simple restart for a migration. The database schema changes require the migrate step.

Metadata Database Backup and Restore

The metadata database holds all DAG runs, task instances, variables, connections, and configuration. Without it, you lose execution history and scheduling state. Backing it up is essential.

PostgreSQL example

Assume your Airflow uses PostgreSQL. You can use pg_dump while the database is online, but for a consistent backup, it is best to either stop the scheduler or use pg_dump with --no-owner and --clean options carefully.

For a consistent snapshot without stopping Airflow, you can use pg_dump with --serializable-deferrable if your PostgreSQL supports it:

pg_dump -h postgres -U airflow -d airflow --serializable-deferrable -Fc -f /backup/airflow/airflow-metadata-$(date +%Y%m%d).dump

This creates a custom-format dump. Verify the dump file exists and is non-zero:

ls -lh /backup/airflow/airflow-metadata-*.dump

Expected size depends on your data volume, but it should not be zero.

To restore:

pg_restore -h postgres -U airflow -d airflow --clean --if-exists /backup/airflow/airflow-metadata-20250101.dump

After restore, verify the database is accessible:

airflow db check

Expected output:

Connection successful.

Then check the database schema matches the Airflow version:

airflow db migrate

This command will apply any pending migrations, but after a restore it should say no migrations needed if the backup was from the same version. If there are pending migrations, run them and verify.

MySQL example

For MySQL, use mysqldump:

mysqldump -h mysql -u airflow -p airflow > /backup/airflow/airflow-metadata-$(date +%Y%m%d).sql

To restore:

mysql -h mysql -u airflow -p airflow < /backup/airflow/airflow-metadata-20250101.sql

Always test restore on a separate instance before relying on it.

DAG Files and Plugins Backup

DAG files define your workflows. They are usually Python scripts stored in the dags_folder. Find the location:

airflow config get-value core dags_folder

Example output:

/opt/airflow/dags

Back up the entire folder, including subdirectories:

tar -czf /backup/airflow/dags-$(date +%Y%m%d).tar.gz -C /opt/airflow/dags .

Plugins are stored in the plugins folder:

airflow config get-value core plugins_folder

Back them up similarly:

tar -czf /backup/airflow/plugins-$(date +%Y%m%d).tar.gz -C /opt/airflow/plugins .

If you store DAGs in Git, ensure the repository is pushed to a remote. A simple backup is a git bundle:

git bundle create /backup/airflow/dags-repo-$(date +%Y%m%d).bundle --all

You can restore with git clone from the bundle.

Logs and Other Files

Airflow logs are essential for debugging, but they can be large. Decide on a retention policy. Back up logs only if required for audit or compliance. They are often stored in AIRFLOW_HOME/logs or in a remote location like S3 if configured.

Example backup of local logs:

tar -czf /backup/airflow/logs-$(date +%Y%m%d).tar.gz -C /opt/airflow/logs .

If logs are sent to S3, no local backup is needed, but you may want to enable versioning on the bucket.

Also back up your airflow.cfg, environment variables file, and any custom scripts.

Verification and Diagnostics After Restore

A backup is only good if you can restore from it. Test your restore process regularly. Here is a checklist:

1. Restore metadata database to a test instance.
2. Restore DAGs and plugins.
3. Start Airflow services.
4. Run health checks.
5. Check DAG parsing: airflow dags list should show your DAGs.
6. Check that the scheduler is responsive: airflow jobs check --job-type SchedulerJob --hostname $(hostname)
7. Trigger a simple DAG run and verify it completes.

Example: after restore, run:

airflow dags list

Expected output includes your DAG IDs, for example:

dag_id              | filepath            | owner   | paused
====================+=====================+=========+=======
example_dag_basic   | example_dag_basic.py| airflow | False

If DAGs are missing, check the DAG folder path and permissions.

Use airflow tasks test for a quick manual test without scheduler:

airflow tasks test example_dag_basic print_date 2025-01-01

This runs the task locally and shows logs. If it fails, the DAG or dependencies are not restored correctly.

Failure Modes and Recovery

Several things can go wrong during backup and restore. Here are common failures and how to handle them.

Failure: Inconsistent metadata database backup

If you take a backup while the scheduler is actively writing, you might get an inconsistent dump. This can lead to missing task instances or corrupted state.

Recovery: Always use a consistent backup method. For PostgreSQL, use pg_dump with --serializable-deferrable, or stop the scheduler temporarily. For MySQL, use mysqldump --single-transaction if using InnoDB. If you already have an inconsistent backup, you may need to re-take it after ensuring consistency.

Failure: DAG files missing after restore

If DAGs are not present after restore, the scheduler will not schedule any runs. Check the dags_folder path and permissions.

Example error in scheduler logs:

FileNotFoundError: [Errno 2] No such file or directory: '/opt/airflow/dags/my_dag.py'

Recovery: Ensure the DAG backup was extracted to the correct location and the Airflow user has read permissions. Verify with ls -l and airflow dags list.

Failure: Database schema mismatch

If the metadata database backup is from an older version, you may need to run airflow db migrate. If the backup is from a newer version, it may not be compatible.

Recovery: Check Airflow version and database version. Run airflow db migrate if needed, but do not downgrade. If incompatible, you may need to use an intermediate version or rebuild the database from DAG definitions.

Failure: Restore succeeds but scheduler does not start

Sometimes the scheduler fails because the metadata database is reachable but some configuration is wrong.

Check scheduler logs for errors. Common issue: wrong database connection string. Verify with airflow db check. If connection fails, fix the connection string in airflow.cfg or environment variable and restart.

Failure: Secrets or connections are missing after restore

Connections and variables are stored in the metadata database, so they should be restored with it. But if you recreated the database from scratch, connections and variables may be lost.

Recovery: Export connections and variables regularly using airflow connections export and airflow variables export. For example:

airflow connections export /backup/airflow/connections.json
airflow variables export /backup/airflow/variables.json

Then you can import them after a fresh database setup:

airflow connections import /backup/airflow/connections.json
airflow variables import /backup/airflow/variables.json

Operations Checklist for Backup and Restore

Here is a concrete checklist to follow. Assign an owner to each item and set a review frequency.

The backup scripts should be stored in version control and reviewed. Alerts should be set up for backup failures.

Example backup script for PostgreSQL metadata DB, DAGs, and logs:

#!/bin/bash
set -euo pipefail
BACKUP_DIR=/backup/airflow
DATE=$(date +%Y%m%d)
mkdir -p $BACKUP_DIR
pg_dump -h postgres -U airflow -d airflow --serializable-deferrable -Fc -f $BACKUP_DIR/airflow-metadata-$DATE.dump
tar -czf $BACKUP_DIR/dags-$DATE.tar.gz -C /opt/airflow/dags .
tar -czf $BACKUP_DIR/logs-$DATE.tar.gz -C /opt/airflow/logs .
# Optionally push to remote storage
# aws s3 cp $BACKUP_DIR s3://my-airflow-backups/$DATE --recursive

Run this script daily via cron or a scheduler.

Common Pitfalls and How to Avoid Them

1. Not testing restores: Many teams back up but never test restoration. Pitfall: when disaster strikes, the backup is unusable. Avoid by scheduling monthly restore tests in a staging environment.

1. Backing up without consistency: Taking a plain pg_dump while Airflow is writing can lead to inconsistent snapshots. Use consistent options or pause the scheduler.

1. Ignoring configuration and secrets: The metadata database is not enough if you have custom airflow.cfg or environment variables. Back them up and encrypt secrets.

1. Forgetting DAG dependencies: DAGs may import custom modules or plugins. If those are not backed up, DAGs fail after restore. Back up the entire plugins folder and any custom libraries.

1. Not versioning backups: Keep multiple backup versions to allow point-in-time recovery. Use a retention policy, e.g., keep daily backups for 30 days and monthly for 12 months.

1. Assuming Airflow 3 and 2 are the same: Airflow 3 changed architecture significantly. Do not copy Airflow 2 backup procedures blindly. Verify component health via the API for Airflow 3.

1. Leaving secrets in plain text: Connection strings in config dumps can expose passwords. Redact or encrypt backups.

Conclusion

Backing up and restoring Apache Airflow requires attention to the metadata database, DAG files, configuration, and secrets. By following the practical commands and checks in this article, you can ensure that your Airflow instance can be recovered quickly and reliably.

Start with a full inventory of your Airflow setup, implement the backup script, and then practice a restore. Validate every step with health checks and DAG listing. Assign clear ownership and review the process regularly.

Your backup is only as good as your last successful restore test. Make that test a routine part of your operations.