Apache Airflow production operations checklist with practical examples (Airflow 2.7+ and 3.x)

Introduction

Treat Apache Airflow like any other critical service: configure it deliberately, observe it continuously, and practice recovery. This operator-focused checklist is scoped for Airflow 2.7+ and 3.x (with 3.3 as the current reference). It shows what to inventory, how to verify component health, what to measure, and how to back up, upgrade, and recover safely. SQLite and standalone or quick-start modes are not production deployments; use an external PostgreSQL or MySQL-compatible metadata database and production-ready executors.

1) Inventory your deployment first

Before tuning or troubleshooting, capture a single source of truth.

Run these commands from a scheduler or utility pod/host.
Replace placeholders like <scheduler-pod> and <database-host> with your values.

Table 1: Version and deployment inventory

Item	Command or source	Example value	Verification step
Airflow version	airflow version	3.3.0	Confirm major.minor and note 2.x vs 3.x features.
Deployment method	docs/runbook	Kubernetes Helm release <release-name>	Match to CI/CD manifest and image digests.
Executor	airflow config get-value core executor	KubernetesExecutor	Ensure executor aligns with worker topology.
Metadata database	airflow config get-value database sql_alchemy_conn	postgresql+psycopg2://...@<database-host>:5432/airflow	Confirm external Postgres or MySQL, not SQLite.
DAG distribution	docs/runbook	Git-sync sidecar to /opt/airflow/dags	Verify DAGs update path and revision.
Remote logging	airflow config get-value logging remote_logging	True	Confirm remote_base_log_folder is reachable.
Secrets backend	airflow config get-value secrets backend	airflow.providers.hashicorp.vault.secrets.vault.VaultBackend	Validate read access with a test connection.
Providers	python -m pip list	apache-airflow-providers-amazon 9.x	Check provider versions against constraints.
Authentication manager	airflow config get-value webserver auth_backend	airflow.api.auth.backend.session	Verify login flow and roles.
Architecture (2 vs 3)	docs/runbook	Airflow 3 with API server and external DAG processor	Ensure components are deployed accordingly.

2) Components and health checks

Know each component’s responsibility and how to verify it:

API server (Airflow 3) or webserver (Airflow 2): Serves UI and API requests.
Scheduler: Creates DagRuns and TaskInstances; places tasks into queues.
DAG processor: Parses DAG code. Independent process in Airflow 3.
Triggerer: Runs deferrable operators and sensors.
Workers/executor: Execute tasks (Celery, Kubernetes, Local).
Metadata database: System of record for Airflow state.
Remote log store: Long-term task log storage (S3, GCS, Azure, etc.).

Health endpoints and CLI checks (copyable):

Airflow 3.x API server health

curl -sSf http://<api-host>/api/v2/monitor/health

Expect HTTP 200 and JSON with per-component status fields; a 200 alone is not sufficient.

Airflow 2.x webserver health

curl -sSf http://<webserver-host>/health

Expect HTTP 200 and a JSON body indicating healthy components.

Scheduler liveness (2.7+ and 3.x)

airflow jobs check --job-type SchedulerJob --hostname <scheduler-hostname>

Exit code 0 indicates the scheduler heartbeat is recent.

Scheduler health server (if enabled in airflow.cfg)

curl -sSf http://<scheduler-host>:8974/health

Expect HTTP 200 with a minimal status body.

DAG processor (Airflow 3.x) process and backlog

ps aux | grep airflow | grep dag-processor

Expect at least one running process; check logs for parse delays and import errors.

Triggerer

airflow jobs check --job-type TriggererJob --hostname <triggerer-hostname>

Exit code 0 indicates the triggerer is heartbeating.

Database connectivity (example Postgres)

psql "host=<database-host> dbname=airflow user=<user>" -c "select 1;"

Expect a single-row result with 1; investigate latency and connections if slow.

Table 2: Component health and first-response matrix

Component or symptom	Likely causes	First safe check	Example command	Expected result
API server or web UI slow	DB saturation, auth backend latency, host CPU	Check DB connections and CPU	psql ... -c "select count(*) from pg_stat_activity;"	Connections within configured limits.
Scheduler heartbeat stalled	DB outage, deadlock, memory pressure, crash loop	Use jobs check and inspect scheduler logs	airflow jobs check --job-type SchedulerJob	Exit 0 when healthy; logs show steady heartbeats.
DAG processing backlog	Expensive imports, too few parsing processes	Inspect dag-processor logs and metrics	grep -i "processing" $AIRFLOW_HOME/logs/dag_processor/*	Parse duration stable and below target.
Growing queued-task age	Executor capacity, pool saturation, worker crash	Check pools and worker concurrency	airflow pools list	Pool used below limit or adjust as needed.
Worker saturation	Too many parallel tasks, host limits	Compare worker slots vs pod or VM limits	kubectl get pods -l component=worker	Desired count running and ready.
Triggerer lag	Many deferrable tasks, triggerer down	Check triggerer job and logs	airflow jobs check --job-type TriggererJob	Exit 0 and low backlog.
DB errors in UI	Network ACL change, TLS cert issue	Test direct DB connectivity	psql ... -c "select 1;"	Query succeeds with low latency.
Remote log failures	Bad IAM, bucket policy, temp creds expired	Tail task logs for upload errors	grep -i "failed to upload" <task-log>	Errors absent after credential refresh.

3) Observability that prevents surprises

Track symptoms that correlate to incidents and triage quickly. Example thresholds are illustrative only; baseline in your environment first.

Scheduler heartbeat and delay: alert if missing for several heartbeat intervals.
DAG processing time and import errors: sustained increases indicate bottlenecks.
Queued and running task age: long ages mean capacity or pool constraints.
Task failures and retries: spikes point to downstream incidents or code regressions.
Zombie detection: investigate tasks with heartbeats missing but marked running.
Pool saturation: pool usage near limit for long periods blocks critical work.
Executor or worker capacity: watch queued length per queue.
Database latency and connections: persistent slow queries or connection spikes.
Triggerer health: backlog of deferrable tasks and triggerer CPU or memory.
API errors and rate: rising 5xx or auth failures signal upstream problems.
Remote log upload errors: ensure logs are complete and timely.
Host or container saturation: CPU, memory, and disk pressure cause flapping.

4) Operational configuration and capacity

Avoid universal numbers; tune from measurements and load tests.

Executor-dependent capacity: match worker slots to downstream system limits. Use pools to cap shared targets.

# Example: cap data warehouse to 5 concurrent tasks
airflow pools set dw_pool 5 "Data warehouse concurrency"

Verify with: airflow pools list and ensure tasks declare pool="dw_pool".

Concurrency and parsing (illustrative)

airflow config get-value core parallelism
airflow config get-value core max_active_tasks_per_dag
airflow config get-value scheduler parsing_processes

Adjust gradually while monitoring DB load and scheduler delay.

Remote logging and environment variables: set logging.remote_logging=True and provide a stable remote_base_log_folder. Keep credentials out of airflow.cfg by referencing a secrets backend. Validate by running a small task and ensuring logs appear remotely within minutes.

Do not use SQLite or standalone/quick-start in production. Use managed PostgreSQL or MySQL-compatible services with backups, connection limits, and TLS.

5) DAG reliability patterns

Idempotent and retry-safe tasks: design tasks so retries do not duplicate side effects. Prefer upserts, partitioned writes, or external transactions.
Stable data intervals: use data intervals and logical dates rather than wall-clock assumptions; avoid time-of-day logic in operators.
Bounded retries and timeouts: set retries, retry_delay, and execution_timeout with pools to prevent runaway work.
Import tests: catch parse-time failures before deployment.

# Detect import errors without running tasks
airflow dags list-import-errors

Expect no errors listed. If errors appear, fix and re-run before enabling the DAG.

6) Security essentials

Least privilege: scope DB, object store, and provider access to the minimum needed.
Secrets backends: store connections and variables in Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault. Rotate regularly.
Encryption: set a valid Fernet key; use TLS for UI and database transports.
Authentication and authorization: enable RBAC with roles aligned to operational duties. For Airflow 3, validate your authentication manager with the API server.
Network boundaries: isolate UI and metadata database within trusted networks.
Auditability: retain UI action logs and provider change history; verify image and provider package provenance.
Avoid shared secrets across components that do not require them.

7) Backup and recovery

Back up assets you can actually restore. Restore drills should be scripted and repeatable.

Table 3: Backup, restore, and validation matrix

Asset	What to back up	Restore approach	Validation step	Notes
Metadata database	Daily full and frequent incrementals	Provision new DB, restore snapshot, point Airflow to it	Airflow starts, DAGs and connections visible	Do not treat running pods or containers as a backup.
DAG source or DAG bundles	Git repository and built artifacts	Re-deploy from tagged commit or artifact	DAG parse succeeds and schedules match	Keep reproducible builds.
Configuration-as-code	airflow.cfg, environment, Helm values	Reapply config to new environment	health checks succeed	Exclude secrets from repos; reference backends.
Secrets references	Secret paths and access policies	Recreate roles or service accounts	Connection test DAG succeeds	Never export raw secrets in backups.
Plugins, providers, custom images	Image registry and provider pin set	Re-tag and redeploy image versions	Import tests pass	Verify image digests match.
Remote logs	Object storage path and lifecycle rules	Point new Airflow to same bucket	Historical task logs retrievable	Retention duration driven by requirements.
Infrastructure definitions	Terraform or cloud templates	Recreate base infra and networking	New environment passes smoke tests	Document per-region differences.

Minimal restore drill (illustrative):

Stand up an empty environment with the same Airflow version and executors.
Restore metadata DB to a new instance; point Airflow to it.
Deploy the same DAGs and provider images.
Run a read-only verification DAG that lists connections and checks one historical run per critical DAG.
Document any gaps; update runbooks accordingly. Recovery is bounded by DB restore point and log retention.

8) Upgrades you can trust

Plan upgrades as controlled changes with tested fallbacks. For Airflow 2.7+ and 3.x, the database command is airflow db migrate.

Table 4: Upgrade stages, gates, and fallback decisions

Stage	Operator actions	Gate to proceed	Fallback if gate fails	Version notes
Plan and constraints	Read release notes and constraints, pin provider versions	Compatibility matrix reviewed	Defer upgrade and open issues	For 3.x, confirm auth and plugin compatibility.
Staging build	Rebuild image, update providers, apply config changes	DAGs import and basic runs succeed	Revert image tag and provider pins	Use the same executor and DB engine as prod.
DB migrate dry run	Stop schedulers, run airflow db migrate in staging	Migrations complete without errors	Restore DB from snapshot and retry	Do not use airflow db upgrade.
Roll or stop-the-world	Choose strategy based on executor and orchestration	Runbook approved	Abort and hold current prod	Rolling is limited for some executors.
Production migrate	Backup, stop schedulers, run airflow db migrate	Airflow starts and health checks pass	Restore DB and previous images	Verify jobs check and API health immediately.
Post-upgrade verification	Run representative DAG tests and review metrics	No regressions over agreed window	Roll back image and DB snapshot	Keep fallback plan valid for the window.

Airflow 3 changes relevant to operators:

API server replaces the 2.x RBAC webserver for API operations.
DAG processor runs independently; scale parsing separately from scheduling.
Task SDK and security boundary tighten imports and execution; validate custom operators and plugins.
Configuration migration checks at startup may block invalid legacy keys; review logs.
Authentication and plugin interfaces can require updates; stage and test thoroughly.

9) Short incident sequence: scheduler heartbeat loss or rising queued-task age

Observe: Alert fires for missing scheduler heartbeat or queued-task age beyond hypothesis threshold.
Scope: Identify affected DAGs, pools, and queues; check if the triggerer or API server is also degraded.
Preserve evidence: Snapshot scheduler and worker logs, note DB connection counts, and record current Airflow version and image digests.
Check dependencies: Verify database health, object store availability, and cluster autoscaling events.
Least risky intervention: Restart only the affected component (for example a single scheduler pod), avoiding wide clears or mass restarts.
Verify recovery: Confirm airflow jobs check returns exit 0, queued age declines, and new tasks are scheduled.
Document follow-up: File a root-cause ticket with metrics, logs, and recommended configuration or capacity changes.

If you consider airflow tasks clear, scope it precisely, dry-run first, and understand downstream effects.

# Dry-run a targeted clear for a single task instance
airflow tasks clear -s 2024-07-01 -e 2024-07-01 <dag-id> <task-id> --dry-run
# Then confirm scope and, if safe, run without --dry-run

Verify the cleared instance reschedules and completes; check for unintended downstream triggers.

10) Routine operations

Create predictable daily, weekly, monthly, and release-time habits.

Table 5: Daily, weekly, monthly, and release-time operations checklist

Cadence	Actions	Verification	Notes
Daily	Review scheduler delay, queued age, top failing DAGs	Dashboards show stable or improving trends	Adjust pools or concurrency based on evidence.
Weekly	Prune old XComs and failed task logs per policy	Storage growth within budget	Use a maintenance DAG for predictable cleanup.
Monthly	Rotate keys, audit roles, vacuum or analyze DB	Access and performance checks pass	Coordinate with security and DB teams.
Release time	Stage, migrate with airflow db migrate, verify DAGs	Health checks and smoke tests pass	Keep rollback plan and snapshot at hand.

Conclusion

Production-ready Airflow is the result of disciplined inventory, explicit health checks, focused observability, cautious configuration, and rehearsed recovery. Use this version-scoped checklist to deploy and operate Airflow 2.7+ and 3.x with confidence: verify components independently, measure symptoms that matter, back up assets you can restore, upgrade with gates and fallbacks, and respond to incidents with the least risky actions that restore service. Tune every illustrative example from your own measurements, and keep your runbooks current as your environment evolves.