Apache Airflow production operations checklist with practical examples: practical implementation guide

Apache Airflow becomes reliable in production when you treat it like any other critical service: configure it deliberately, measure it, and practice recovery. This checklist gives you practical steps and copyable examples to harden Airflow across configuration, monitoring, maintenance, backups, upgrades, security, and performance.

Why a checklist? It turns a complex topic into clear, reviewable steps you can apply in your environment. Breaking work into stages reduces rework and outages, and a small pilot lets you prove safety before scaling.

Workflow Overview

Use a simple, staged flow for Airflow operations:

1. Plan

• Define availability targets (SLOs) for the scheduler and critical DAGs.
• Document data dependencies and external system quotas.

1. Configure

• Apply hardened defaults in airflow.cfg and environment variables.
• Create pools, connections, variables, and secrets.

1. Validate

• Parse DAGs non-interactively and fail fast on import errors.
• Dry-run critical DAGs and prove idempotency of tasks.

1. Observe

• Emit metrics, centralize logs, and set alerts with clear thresholds.
• Watch scheduler and worker saturation, DAG parse time, and task queues.

1. Maintain

• Rotate keys, prune old logs and XComs, archive historical runs.
• Run periodic verification DAGs to test connectivity and alerts.

1. Improve

• Triage failure patterns and slow DAGs.
• Adjust pools, retries, and schedules based on evidence.

Production Configuration Checklist

Apply these settings before you scale up. Example snippets are illustrative; adjust for your environment.

Core settings (airflow.cfg)

[core]
executor = CeleryExecutor            # or KubernetesExecutor
load_examples = False
fernet_key = YOUR_32_BYTE_BASE64_KEY
sql_alchemy_conn = postgresql+psycopg2://airflow:*****@db:5432/airflow
parallelism = 64                     # global task slots
max_active_tasks_per_dag = 16
max_active_runs_per_dag = 1

[logging]
remote_logging = True
remote_base_log_folder = s3://your-bucket/airflow-logs
logging_level = INFO

[scheduler]
min_file_process_interval = 30
max_tis_per_query = 256
scheduler_heartbeat_sec = 5
parsing_processes = 4

[metrics]
statsd_on = True
statsd_host = metrics.local
statsd_port = 8125

[email]
email_backend = airflow.utils.email.send_email_smtp

[webserver]
rbac = True
expose_config = False

Tips:

• Set fernet_key and use a secrets backend for connections and variables.
• Keep max_active_runs_per_dag low for heavy DAGs to avoid thundering herds.
• Use remote_logging to centralize task logs.

Pools and queues

Pools protect shared systems from overload.

# Example: limit queries to a data warehouse to 5 concurrent tasks
airflow pools set dw_pool 5 "Data warehouse concurrency"

Use pools in tasks:

PythonOperator(
    task_id="load_dw",
    python_callable=load_fn,
    pool="dw_pool",
)

Sensible DAG defaults

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator

default_args = {
    "owner": "data-eng",
    "depends_on_past": False,
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
    "email_on_failure": True,
    "email": ["[email protected]"],
}

dag = DAG(
    dag_id="example_daily_etl",
    default_args=default_args,
    start_date=datetime(2024, 1, 1),
    schedule_interval="0 2 * * *",
    catchup=False,
    max_active_runs=1,
    tags=["etl"],
)

PythonOperator(
    task_id="extract",
    python_callable=extract_fn,
    dag=dag,
    pool="dw_pool",
)

Database reliability

• Use a managed Postgres-compatible database with automated backups and connection limits.
• Set connection pool sizes and health checks.
• Keep airflow metadata DB isolated from heavy analytics workloads.

Monitoring and Alerting

Measure what matters and alert on actionable symptoms.

Health checks

• Webserver and scheduler health endpoints: probe every 30s.
• Alert if scheduler heartbeat stops, or if queued tasks age beyond SLO.

Metrics to collect

• Scheduler: dag_parsing.errors, dag.parsing.time, scheduler.heartbeat.
• Task lifecycle: tasks.queued, tasks.running, tasks.failed, tasks.duration.
• Executor/worker saturation: pool usage, worker concurrency, queue length.
• DAG outcomes: success rate by DAG, retry counts, SLA misses.

Example thresholds:

• Alert if tasks.failed for a DAG exceed 3 in 15 minutes.
• Alert if queued task age > 10 minutes for critical DAGs.
• Alert if dag.parsing.time p95 > 2s or import errors increase.

Centralized logs

Keep 14 to 30 days of hot logs; archive older logs to object storage. Ensure logs include run_id, try_number, and task_id for easy correlation.

Alert routing

• Route critical DAG alerts to on-call with paging.
• Route non-critical to chat or email.
• Include run_id, upstream dependency status, and suggested remediation steps in alerts.

Operations and Maintenance

Establish predictable routines.

Weekly

• [ ] Review top failing DAGs and top retrying tasks.
• [ ] Prune XComs older than 30 days.
• [ ] Verify pools reflect current downstream limits.
• [ ] Check DAG parse errors and import times.

Monthly

• [ ] Rotate service account keys and fernet_key if policy requires.
• [ ] Archive logs older than retention window.
• [ ] Run database vacuum/analyze where applicable.
• [ ] Audit Airflow roles and user access.

Useful commands

# Parse DAGs without running to catch import errors
airflow dags list-import-errors

# Clear and rerun a specific task instance safely (example)
airflow tasks clear -s 2024-07-01 -e 2024-07-01 example_daily_etl load --reset-dagruns

# XCom cleanup via maintenance job (example custom script)
python cleanup_xcoms.py --older-than-days 30

Backups and Disaster Recovery

Back up three things: metadata DB, DAG code, and logs. Test restores.

Backup plan

• Metadata DB: full daily, WAL or incremental every 5 to 15 minutes.
• DAGs repository: version control plus nightly artifact snapshot.
• Logs: archive to object storage with lifecycle rules.

Example Postgres backup

pg_dump -Fc -h db -U airflow airflow > airflow_$(date +%F).dump

Recovery drills

• Quarterly: restore metadata DB to a fresh instance and point a test Airflow at it.
• Verify DAGs and connections load, and that backfill/replay works as expected.
• Define RPO and RTO; ensure backups meet these targets.

Upgrades and Change Management

Treat upgrades as planned changes with safety rails.

Checklist

• [ ] Read release notes for core and provider packages.
• [ ] Pin versions; upgrade providers with Airflow when possible.
• [ ] Run database migrations in a staging environment first.
• [ ] Validate DAG imports and dry-run critical DAGs.
• [ ] Back up metadata DB and DAGs before upgrade.
• [ ] Have a rollback plan (package versions and DB snapshot).

Example sequence

1. Freeze current versions and back up DB.
2. Upgrade Airflow packages.
3. Run airflow db upgrade.
4. Start scheduler only; watch logs and metrics.
5. Start workers; monitor queued tasks and failures.
6. Promote if stable; otherwise roll back.

Security and Access Control

Reduce exposure and protect credentials.

• Enable RBAC and use least-privilege roles.
• Disable expose_config and restrict webserver to trusted networks.
• Use a secrets backend for connections and variables; avoid plain-text.
• Enforce TLS for UI and metadata DB.
• Log user actions in the UI for audit.
• Validate DAG inputs; never trust external parameters without checks.

Performance and Scaling

Tune to prevent backlogs and flapping.

Scheduler

• Increase max_tis_per_query to reduce DB round trips (test carefully).
• Keep parsing_processes aligned with CPU limits.
• Watch dag.parsing.time and import errors.

Workers and executors

• Right-size worker concurrency; avoid exhausting downstream systems.
• Use pools for shared services; adjust pool sizes based on error and retry rates.
• Prefer deferrable operators for long waits to free worker slots where available.

DAG design

• Avoid generating thousands of tasks per run; batch when possible.
• Cap backfill with max_active_runs_per_dag and start small.
• Set retries and retry_delay based on downstream SLAs.

Database

• Use connection pooling; monitor connection count and slow queries.
• Vacuum/analyze periodically to keep query plans stable.

Common Production Pitfalls and Fixes

Problem: Catchup storms after deploy

• Cause: New DAG with old start_date and catchup=True.
• Fix: Set catchup=False for streaming/nearline jobs; if backfill is needed, run in controlled windows.

Problem: XCom bloat

• Cause: Large payloads pushed to XCom.
• Fix: Store large objects externally and pass references; prune old XComs.

Problem: Dynamic task explosion

• Cause: Overly granular task mapping.
• Fix: Batch work into chunks; use pools and max_active_tasks_per_dag.

Problem: Slow DAG parsing

• Cause: Heavy imports or network calls at parse time.
• Fix: Move IO to operator execution; cache configuration; keep parse-time code light.

Problem: Flaky sensors and timeouts

• Cause: Short poke intervals or tight timeouts.
• Fix: Increase poke_interval; use deferrable sensors where available; set sensible timeouts plus retries.

Problem: Misfires and schedule drift

• Cause: Long task durations exceed schedule interval.
• Fix: Increase schedule interval or split tasks; set max_active_runs_per_dag=1.

Local Pilot Plan

Start with a single, low-risk DAG and a narrow, measurable goal.

Pilot goal

• Prove that monitoring, alerts, and basic recovery work for one critical path.

Scope

• One DAG that runs daily, touches a non-critical data source, and has 2 to 3 tasks.

Steps

1. Implement DAG with pools, retries, and email_on_failure.
2. Add metrics and confirm they appear in your dashboard.
3. Trigger success and a controlled failure; confirm alerts with run_id and task_id.
4. Practice recovery: clear a failed task and rerun; verify idempotency.
5. Record results and adjust thresholds or pool sizes.

Success criteria

• Alerts fire within 2 minutes of failure.
• Median task duration within target; no queue buildup.
• Recovery time from failure under agreed SLO.

Conclusion

Reliable Airflow operations come from disciplined configuration, visible health signals, routine maintenance, and practiced recovery. Use this checklist to harden core settings, monitor the right metrics, back up the pieces that matter, and tune for steady flow. Begin with a narrow pilot, measure the outcomes, then scale the same practices across your DAGs.