## Introduction

Treat Apache Airflow like any other critical service: configure it deliberately, observe it continuously, and practice recovery. This operator-focused checklist is scoped for Airflow 2.7+ and 3.x (with 3.3 as the current reference). It shows what to inventory, how to verify component health, what to measure, and how to back up, upgrade, and recover safely. SQLite and standalone or quick-start modes are not production deployments; use an external PostgreSQL or MySQL-compatible metadata database and production-ready executors.

## 1) Inventory your deployment first

Before tuning or troubleshooting, capture a single source of truth.

- Run these commands from a scheduler or utility pod/host.

- Replace placeholders like <scheduler-pod> and <database-host> with your values.

Table 1: Version and deployment inventory

<div class="my-stack-md overflow-x-auto">
<table class="w-full min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Item</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Command or source</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Example value</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Verification step</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Airflow version</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">airflow version</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3.3.0</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Confirm major.minor and note 2.x vs 3.x features.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Deployment method</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">docs/runbook</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Kubernetes Helm release &lt;release-name&gt;</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Match to CI/CD manifest and image digests.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Executor</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">airflow config get-value core executor</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">KubernetesExecutor</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Ensure executor aligns with worker topology.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Metadata database</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">airflow config get-value database sql_alchemy_conn</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">postgresql+psycopg2://...@&lt;database-host&gt;:5432/airflow</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Confirm external Postgres or MySQL, not SQLite.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">DAG distribution</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">docs/runbook</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Git-sync sidecar to /opt/airflow/dags</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Verify DAGs update path and revision.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Remote logging</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">airflow config get-value logging remote_logging</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">True</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Confirm remote_base_log_folder is reachable.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Secrets backend</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">airflow config get-value secrets backend</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">airflow.providers.hashicorp.vault.secrets.vault.VaultBackend</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Validate read access with a test connection.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Providers</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">python -m pip list</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">apache-airflow-providers-amazon 9.x</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Check provider versions against constraints.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Authentication manager</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">airflow config get-value webserver auth_backend</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">airflow.api.auth.backend.session</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Verify login flow and roles.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Architecture (2 vs 3)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">docs/runbook</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Airflow 3 with API server and external DAG processor</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Ensure components are deployed accordingly.</td></tr></tbody>
</table>
</div>

## 2) Components and health checks

Know each component’s responsibility and how to verify it:

- API server (Airflow 3) or webserver (Airflow 2): Serves UI and API requests.

- Scheduler: Creates DagRuns and TaskInstances; places tasks into queues.

- DAG processor: Parses DAG code. Independent process in Airflow 3.

- Triggerer: Runs deferrable operators and sensors.

- Workers/executor: Execute tasks (Celery, Kubernetes, Local).

- Metadata database: System of record for Airflow state.

- Remote log store: Long-term task log storage (S3, GCS, Azure, etc.).

Health endpoints and CLI checks (copyable):

- Airflow 3.x API server health

curl -sSf http://<api-host>/api/v2/monitor/health 
 Expect HTTP 200 and JSON with per-component status fields; a 200 alone is not sufficient.

- Airflow 2.x webserver health

curl -sSf http://<webserver-host>/health 
 Expect HTTP 200 and a JSON body indicating healthy components.

- Scheduler liveness (2.7+ and 3.x)

airflow jobs check --job-type SchedulerJob --hostname <scheduler-hostname> 
 Exit code 0 indicates the scheduler heartbeat is recent.

- Scheduler health server (if enabled in airflow.cfg)

curl -sSf http://<scheduler-host>:8974/health 
 Expect HTTP 200 with a minimal status body.

- DAG processor (Airflow 3.x) process and backlog

ps aux | grep airflow | grep dag-processor 
 Expect at least one running process; check logs for parse delays and import errors.

- Triggerer

airflow jobs check --job-type TriggererJob --hostname <triggerer-hostname> 
 Exit code 0 indicates the triggerer is heartbeating.

- Database connectivity (example Postgres)

psql "host=<database-host> dbname=airflow user=<user>" -c "select 1;" 
 Expect a single-row result with 1; investigate latency and connections if slow.

Table 2: Component health and first-response matrix

<div class="my-stack-md overflow-x-auto">
<table class="w-full min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Component or symptom</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Likely causes</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">First safe check</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Example command</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Expected result</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">API server or web UI slow</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">DB saturation, auth backend latency, host CPU</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Check DB connections and CPU</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">psql ... -c &quot;select count(*) from pg_stat_activity;&quot;</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Connections within configured limits.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Scheduler heartbeat stalled</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">DB outage, deadlock, memory pressure, crash loop</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Use jobs check and inspect scheduler logs</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">airflow jobs check --job-type SchedulerJob</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Exit 0 when healthy; logs show steady heartbeats.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">DAG processing backlog</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Expensive imports, too few parsing processes</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Inspect dag-processor logs and metrics</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">grep -i &quot;processing&quot; $AIRFLOW_HOME/logs/dag_processor/*</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Parse duration stable and below target.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Growing queued-task age</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Executor capacity, pool saturation, worker crash</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Check pools and worker concurrency</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">airflow pools list</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Pool used below limit or adjust as needed.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Worker saturation</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Too many parallel tasks, host limits</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Compare worker slots vs pod or VM limits</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">kubectl get pods -l component=worker</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Desired count running and ready.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Triggerer lag</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Many deferrable tasks, triggerer down</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Check triggerer job and logs</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">airflow jobs check --job-type TriggererJob</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Exit 0 and low backlog.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">DB errors in UI</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Network ACL change, TLS cert issue</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Test direct DB connectivity</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">psql ... -c &quot;select 1;&quot;</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Query succeeds with low latency.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Remote log failures</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Bad IAM, bucket policy, temp creds expired</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Tail task logs for upload errors</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">grep -i &quot;failed to upload&quot; &lt;task-log&gt;</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Errors absent after credential refresh.</td></tr></tbody>
</table>
</div>

## 3) Observability that prevents surprises

Track symptoms that correlate to incidents and triage quickly. Example thresholds are illustrative only; baseline in your environment first.

- Scheduler heartbeat and delay: alert if missing for several heartbeat intervals.

- DAG processing time and import errors: sustained increases indicate bottlenecks.

- Queued and running task age: long ages mean capacity or pool constraints.

- Task failures and retries: spikes point to downstream incidents or code regressions.

- Zombie detection: investigate tasks with heartbeats missing but marked running.

- Pool saturation: pool usage near limit for long periods blocks critical work.

- Executor or worker capacity: watch queued length per queue.

- Database latency and connections: persistent slow queries or connection spikes.

- Triggerer health: backlog of deferrable tasks and triggerer CPU or memory.

- API errors and rate: rising 5xx or auth failures signal upstream problems.

- Remote log upload errors: ensure logs are complete and timely.

- Host or container saturation: CPU, memory, and disk pressure cause flapping.

## 4) Operational configuration and capacity

Avoid universal numbers; tune from measurements and load tests.

- Executor-dependent capacity: match worker slots to downstream system limits. Use pools to cap shared targets.

# Example: cap data warehouse to 5 concurrent tasks
airflow pools set dw_pool 5 "Data warehouse concurrency" 
 Verify with: airflow pools list and ensure tasks declare pool="dw_pool" .

- Concurrency and parsing (illustrative)

airflow config get-value core parallelism
airflow config get-value core max_active_tasks_per_dag
airflow config get-value scheduler parsing_processes 
 Adjust gradually while monitoring DB load and scheduler delay.

- Remote logging and environment variables: set logging.remote_logging=True and provide a stable remote_base_log_folder. Keep credentials out of airflow.cfg by referencing a secrets backend. Validate by running a small task and ensuring logs appear remotely within minutes.

- Do not use SQLite or standalone/quick-start in production. Use managed PostgreSQL or MySQL-compatible services with backups, connection limits, and TLS.

## 5) DAG reliability patterns

- Idempotent and retry-safe tasks: design tasks so retries do not duplicate side effects. Prefer upserts, partitioned writes, or external transactions.

- Stable data intervals: use data intervals and logical dates rather than wall-clock assumptions; avoid time-of-day logic in operators.

- Bounded retries and timeouts: set retries , retry_delay , and execution_timeout with pools to prevent runaway work.

- Import tests: catch parse-time failures before deployment.

# Detect import errors without running tasks
airflow dags list-import-errors 
 Expect no errors listed. If errors appear, fix and re-run before enabling the DAG.

## 6) Security essentials

- Least privilege: scope DB, object store, and provider access to the minimum needed.

- Secrets backends: store connections and variables in Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault. Rotate regularly.

- Encryption: set a valid Fernet key; use TLS for UI and database transports.

- Authentication and authorization: enable RBAC with roles aligned to operational duties. For Airflow 3, validate your authentication manager with the API server.

- Network boundaries: isolate UI and metadata database within trusted networks.

- Auditability: retain UI action logs and provider change history; verify image and provider package provenance.

- Avoid shared secrets across components that do not require them.

## 7) Backup and recovery

Back up assets you can actually restore. Restore drills should be scripted and repeatable.

Table 3: Backup, restore, and validation matrix

<div class="my-stack-md overflow-x-auto">
<table class="w-full min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Asset</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">What to back up</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Restore approach</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Validation step</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Notes</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Metadata database</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Daily full and frequent incrementals</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Provision new DB, restore snapshot, point Airflow to it</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Airflow starts, DAGs and connections visible</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Do not treat running pods or containers as a backup.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">DAG source or DAG bundles</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Git repository and built artifacts</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Re-deploy from tagged commit or artifact</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">DAG parse succeeds and schedules match</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Keep reproducible builds.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Configuration-as-code</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">airflow.cfg, environment, Helm values</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Reapply config to new environment</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">health checks succeed</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Exclude secrets from repos; reference backends.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Secrets references</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Secret paths and access policies</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Recreate roles or service accounts</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Connection test DAG succeeds</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Never export raw secrets in backups.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Plugins, providers, custom images</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Image registry and provider pin set</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Re-tag and redeploy image versions</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Import tests pass</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Verify image digests match.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Remote logs</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Object storage path and lifecycle rules</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Point new Airflow to same bucket</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Historical task logs retrievable</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Retention duration driven by requirements.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Infrastructure definitions</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Terraform or cloud templates</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Recreate base infra and networking</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">New environment passes smoke tests</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Document per-region differences.</td></tr></tbody>
</table>
</div>
Minimal restore drill (illustrative):

- Stand up an empty environment with the same Airflow version and executors.

- Restore metadata DB to a new instance; point Airflow to it.

- Deploy the same DAGs and provider images.

- Run a read-only verification DAG that lists connections and checks one historical run per critical DAG.

- Document any gaps; update runbooks accordingly. Recovery is bounded by DB restore point and log retention.

## 8) Upgrades you can trust

Plan upgrades as controlled changes with tested fallbacks. For Airflow 2.7+ and 3.x, the database command is airflow db migrate .

Table 4: Upgrade stages, gates, and fallback decisions

<div class="my-stack-md overflow-x-auto">
<table class="w-full min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Stage</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Operator actions</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Gate to proceed</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Fallback if gate fails</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Version notes</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Plan and constraints</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Read release notes and constraints, pin provider versions</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Compatibility matrix reviewed</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Defer upgrade and open issues</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">For 3.x, confirm auth and plugin compatibility.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Staging build</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Rebuild image, update providers, apply config changes</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">DAGs import and basic runs succeed</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Revert image tag and provider pins</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Use the same executor and DB engine as prod.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">DB migrate dry run</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Stop schedulers, run airflow db migrate in staging</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Migrations complete without errors</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Restore DB from snapshot and retry</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Do not use airflow db upgrade.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Roll or stop-the-world</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Choose strategy based on executor and orchestration</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Runbook approved</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Abort and hold current prod</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Rolling is limited for some executors.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Production migrate</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Backup, stop schedulers, run airflow db migrate</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Airflow starts and health checks pass</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Restore DB and previous images</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Verify jobs check and API health immediately.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Post-upgrade verification</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Run representative DAG tests and review metrics</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">No regressions over agreed window</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Roll back image and DB snapshot</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Keep fallback plan valid for the window.</td></tr></tbody>
</table>
</div>
Airflow 3 changes relevant to operators:

- API server replaces the 2.x RBAC webserver for API operations.

- DAG processor runs independently; scale parsing separately from scheduling.

- Task SDK and security boundary tighten imports and execution; validate custom operators and plugins.

- Configuration migration checks at startup may block invalid legacy keys; review logs.

- Authentication and plugin interfaces can require updates; stage and test thoroughly.

## 9) Short incident sequence: scheduler heartbeat loss or rising queued-task age

- Observe: Alert fires for missing scheduler heartbeat or queued-task age beyond hypothesis threshold.

- Scope: Identify affected DAGs, pools, and queues; check if the triggerer or API server is also degraded.

- Preserve evidence: Snapshot scheduler and worker logs, note DB connection counts, and record current Airflow version and image digests.

- Check dependencies: Verify database health, object store availability, and cluster autoscaling events.

- Least risky intervention: Restart only the affected component (for example a single scheduler pod), avoiding wide clears or mass restarts.

- Verify recovery: Confirm airflow jobs check returns exit 0, queued age declines, and new tasks are scheduled.

- Document follow-up: File a root-cause ticket with metrics, logs, and recommended configuration or capacity changes.

If you consider airflow tasks clear , scope it precisely, dry-run first, and understand downstream effects.

# Dry-run a targeted clear for a single task instance
airflow tasks clear -s 2024-07-01 -e 2024-07-01 <dag-id> <task-id> --dry-run
# Then confirm scope and, if safe, run without --dry-run 
 Verify the cleared instance reschedules and completes; check for unintended downstream triggers.

## 10) Routine operations

Create predictable daily, weekly, monthly, and release-time habits.

Table 5: Daily, weekly, monthly, and release-time operations checklist

<div class="my-stack-md overflow-x-auto">
<table class="w-full min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Cadence</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Actions</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Verification</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Notes</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Daily</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Review scheduler delay, queued age, top failing DAGs</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Dashboards show stable or improving trends</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Adjust pools or concurrency based on evidence.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Weekly</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Prune old XComs and failed task logs per policy</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Storage growth within budget</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Use a maintenance DAG for predictable cleanup.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Monthly</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Rotate keys, audit roles, vacuum or analyze DB</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Access and performance checks pass</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Coordinate with security and DB teams.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Release time</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Stage, migrate with airflow db migrate, verify DAGs</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Health checks and smoke tests pass</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Keep rollback plan and snapshot at hand.</td></tr></tbody>
</table>
</div>

## Conclusion

Production-ready Airflow is the result of disciplined inventory, explicit health checks, focused observability, cautious configuration, and rehearsed recovery. Use this version-scoped checklist to deploy and operate Airflow 2.7+ and 3.x with confidence: verify components independently, measure symptoms that matter, back up assets you can restore, upgrade with gates and fallbacks, and respond to incidents with the least risky actions that restore service. Tune every illustrative example from your own measurements, and keep your runbooks current as your environment evolves.