Apache Airflow orchestrates critical data and automation workflows. When schedules drift, retries escalate, or the scheduler stalls, you need to know quickly with enough context to act. This guide delivers a practical, safe approach to monitoring and alerts for Airflow. You will: - Identify the first metrics and log signals that matter. - Configure baseline alerts (SLA misses, failures, scheduler health). - Add a compact dashboard for situational awareness. - Verify the setup with controlled tests. - Prepare failure-mode playbooks and rollbacks. The result is a monitoring foundation you can expand confidently. ## Version and Environment Inventory Before making changes, write down a concise inventory so you know exactly where to configure and how to roll back. Use this template and replace values for your environment. Inventory template (fill with your values): - Airflow version: 2.6.x or later recommended - Python version: 3.8+ recommended - Executor: LocalExecutor or CeleryExecutor - Webserver host: host or VIP, port 8080 (adjust as used) - Scheduler host(s): hostnames - Workers: count and hostnames (if using CeleryExecutor) - Metadata DB: engine (e.g., Postgres), host, connection pooling - Message broker: e.g., Redis or RabbitMQ (if CeleryExecutor) - Log storage: local filesystem or remote (e.g., S3 or GCS) - Email SMTP: hostname, from address - Alert channels: email list, Slack webhook URL (if used) - Config file path(s): airflow.cfg and any environment-variable overrides - Access: shell account for Airflow service user and file backup location Prerequisites: - Administrative access to update airflow.cfg or environment variables. - Access to Airflow web UI and API (for /health checks). - A notification channel (email SMTP or a webhook) you can safely test. - Optionally: a metrics sink (e.g., StatsD-compatible endpoint or Prometheus collector) and a dashboard tool. ## Safe Configuration Path Scope the initial rollout to a small, verifiable pilot. Use one representative DAG to prove the loop end-to-end before scaling out. ### Pilot scope - One DAG with 3-5 tasks that runs at least hourly so you can iterate quickly. - Baseline signals: scheduler health, DAG run duration, task failures, and queue/backlog. - Two alert rules to start: scheduler unhealthy, and a spike in task failures for the pilot DAG. ### 1. Enable baseline email and metrics in Airflow Add or confirm these airflow.cfg entries (create a dated backup first): # airflow.cfg excerpts [email] email_backend = airflow.utils.email.send_email_smtp smtp_host = smtp.example.com smtp_starttls = True smtp_ssl = False smtp_user = airflow@example.com smtp_password = YOUR_APP_PASSWORD smtp_mail_from = airflow@example.com [metrics] # Enable StatsD if you have a StatsD-compatible sink. statsd_on = True statsd_host = 127.0.0.1 statsd_port = 8125 statsd_prefix = airflow # Emit hostname in tags if supported by your metrics sink. [logging] # Ensure remote or local logs are accessible and retained. base_log_folder = /var/log/airflow [core] # Keep a modest parallelism to avoid noisy spikes during pilot. parallelism = 16 Environment variable equivalents (if you prefer env-based configuration): AIRFLOW__EMAIL__EMAIL_BACKEND=airflow.utils.email.send_email_smtp AIRFLOW__EMAIL__SMTP_HOST=smtp.example.com AIRFLOW__EMAIL__SMTP_STARTTLS=True AIRFLOW__EMAIL__SMTP_SSL=False AIRFLOW__EMAIL__SMTP_USER=airflow@example.com AIRFLOW__EMAIL__SMTP_PASSWORD=YOUR_APP_PASSWORD AIRFLOW__EMAIL__SMTP_MAIL_FROM=airflow@example.com AIRFLOW__METRICS__STATSD_ON=True AIRFLOW__METRICS__STATSD_HOST=127.0.0.1 AIRFLOW__METRICS__STATSD_PORT=8125 AIRFLOW__METRICS__STATSD_PREFIX=airflow Restart Airflow services in a maintenance window appropriate for your environment. ### 2. Add a failure callback and SLA to a pilot DAG This example shows a DAG-level on_failure_callback that posts to a webhook (e.g., a Slack Incoming Webhook). Adjust the URL and message fields to your channel. The code is intentionally simple for pilot purposes. # dags/pilot_monitoring_example.py from airflow import DAG from airflow.operators.python import PythonOperator from airflow.utils.dates import days_ago from datetime import timedelta import json import urllib.request WEBHOOK_URL = "https://hooks.example.com/services/T000/B000/XXXXX" # replace def notify(context): dag_id = context.get("dag_run\u200d).dag_id if context.get("dag_run‍) else context.get("dag‍).dag_id task_id = context.get("task_instance‍).task_id run_id = context.get("dag_run‍).run_id if context.get("dag_run‍) else "manual__test" text = f"Airflow alert: {dag_id}.{task_id} failed on run {run_id}" data = json.dumps({"text": text}).encode("utf-8\u200d) req = urllib.request.Request(WEBHOOK_URL, data=data, headers={"Content-Type": "application/json"‍) try: urllib.request.urlopen(req, timeout=5) except Exception: pass # Do not fail the task due to notification issues with DAG( dag_id="pilot_monitoring_example", start_date=days_ago(1), schedule_interval="@hourly", catchup=False, sla_miss_callback=notify, # alert on SLA misses default_args={ "email_on_failure": True, "email_on_retry": False, "email": ["airflow-ops@example.com"], "on_failure_callback": notify, "retries": 1, "retry_delay": timedelta(minutes=5), "sla": timedelta(minutes=20), }, tags=["monitoring-pilot"], ) as dag: def ok_task(): return "ok" def flaky_task(): raise RuntimeError("constructed failure for pilot test‍) t1 = PythonOperator(task_id="ok", python_callable=ok_task) t2 = PythonOperator(task_id="flaky", python_callable=flaky_task) t1 >> t2 Notes: - The webhook and email are both used, so you can compare alert timeliness and reliability. - SLA on default_args means each task has a 20-minute SLA; adjust to your DAG characteristics. ### 3. Pipe metrics to a collector If your collector supports StatsD, point it to the host and port above. For Prometheus via statsd_exporter, a mapping file might look like this: mappings: - match: "airflow.scheduler.heartbeat" name: "airflow_scheduler_heartbeat_count" labels: {} - match: "airflow.dag_processing.import_errors" name: "airflow_dag_import_errors_total" - match: "airflow.operator_successes" name: "airflow_task_success_total" labels: operator: "${1}‍ - match: "airflow.operator_failures" name: "airflow_task_fail_total" labels: operator: "${1}‍ ### 4. Create a compact dashboard Start with 4-6 panels you can inspect at a glance: - Scheduler heartbeat count and age trend (5m window). - DAG run duration p50/p95 for the pilot DAG. - Task success vs failure count (rate) for the pilot DAG. - Queued tasks and running tasks (platform-wide). - Import errors count (parsing problems). - Optional: database connection pool saturation if exposed by your stack. ### 5. Define two or three targeted alert rules Create alerts that are easy to reason about. Replace metric names and labels to match your collector.
Alert nameSignalExample conditionAction
SchedulerUnhealthyScheduler heartbeat countNo heartbeat increase in 2mPage on-call and auto-run health check
PilotDagFailuresTask failures for pilot DAG>3 failures in 15m windowNotify channel and create ticket
ImportErrorsDAG import errorsAny increase over baselineNotify maintainers with file list
### 6. Document owners and severity - Pilot DAG owner: team or person who can fix failures. - On-call rotation: who pages, when, and escalation. - Severity mapping: for example, scheduler down = SEV-1; single DAG failures = SEV-3. ## What to Monitor First This table prioritizes early, high-signal metrics. Tune thresholds after a week of observations.
Metric or log signalWhy it mattersPilot threshold
Scheduler heartbeat ageDetects scheduler stalls>120s age triggers alert
DAG run duration p95Detects slowdowns and backlogs+50% over 7-day baseline
Task failure rateCatches regressions fast>3 failures/15m for pilot DAG
Queued tasks countReveals capacity/backlog>2x normal for 10m
Import errorsBad DAG code or depsAny nonzero in last 10m
SLA missesUser-facing latency breachesAny SLA miss on pilot DAG
DB connectivity errors in logsPlatform riskAny spike vs baseline
## Verification and Diagnostics Prove the setup with controlled tests and observable outcomes. ### 1. Health check the webserver and scheduler Airflow exposes a basic health endpoint in recent versions: curl -s http://:8080/health | jq . Expected response: { "metadatabase": {"status": "healthy"}, "scheduler": {"status": "healthy"} } If you cannot access the endpoint, use server logs or service supervision tools to confirm processes are running. ### 2. Induce a controlled failure - Unpause the pilot DAG in the UI. - Manually trigger a run (Run -> Trigger DAG) or wait for the schedule. - The task flaky fails by design. Within a minute, you should see: - An email sent to airflow-ops@example.com . - A webhook notification in your channel. - A bump in task failure metrics in your dashboard. ### 3. Verify metrics emission If using statsd_exporter on localhost port 9102: curl -s http://127.0.0.1:9102/metrics | grep -E "airflow_" Expected: several airflow_* series including scheduler, task successes/failures, and import errors. ### 4. Inspect logs for signal quality Open the flaky task log in the UI. Confirm the exception is visible and timestamps align with your alert timestamps. Ensure logs are retained where your operators can read them. ### 5. Validate dashboard panels - Scheduler heartbeat panel: should show regular increments. - DAG run duration p95: should reflect recent runs; the failed run may be shorter but should appear on the timeline. - Task success/failure: visible spike at test time. - Queued tasks: remain near normal if capacity is adequate. ### 6. Diagnose if something did not work - No email : verify SMTP host, credentials, and firewall rules. Try a test email via a simple Python script using the same SMTP settings. - No webhook alert : test the webhook URL with curl; check proxy rules. - No metrics : confirm statsd_on is true, exporter is running, and ports are open. Look for errors in scheduler logs about metrics clients. - Health check fails : restart affected Airflow services in your maintenance procedure and check the metadata DB connectivity. ## Failure Modes and Recovery Common issues you may encounter and how to recover safely.
SymptomLikely causeFirst diagnostic stepRecovery action
Alert noise every hourThreshold too tightCompare to 7-day baselineRaise threshold or add 2x confirmation window
No scheduler heartbeatStalled schedulerCheck /health and scheduler logsRestart scheduler and verify DB health
DAG import errors burstBad code deployInspect import_errors and offending filesRevert DAG commit; clear import errors
Webhook alerts missingNetwork/proxy issueSend test curl to webhookFallback to email; open network ticket
SLA misses overnightUnder-provisioned poolReview pool slots vs queued tasksAdd slots or reschedule DAGs
Metric gap in dashboardExporter downCheck exporter logs and portRestart exporter; backfill from logs if possible
### Rollback playbook - Back up airflow.cfg before changes; to roll back, restore the previous copy and restart services. - If a callback causes failures, remove the on_failure_callback and sla_miss_callback lines from the pilot DAG, redeploy, and clear only the affected DAG runs. - Disable noisy alert rules by toggling them off in your alerting tool, then revisit thresholds. - If the metrics path causes instability, set statsd_on = False and restart to confirm stability. - Always re-run the /health check and a manual pilot DAG run to validate recovery. ### Incident response workflow - Triage within 5 minutes : classify severity; if scheduler unhealthy, declare SEV-1 and page on-call. - Contain : pause high-churn DAGs if needed to reduce load during recovery. - Remediate : restart components as necessary; apply targeted fixes (e.g., revert a faulty DAG file). - Verify : run /health , confirm dashboard normalization, and check new runs of affected DAGs. - Document : note cause, fix, and threshold adjustments for the next noise review. ## Operations Checklist ### Daily - Check Airflow /health endpoint for green status on metadatabase and scheduler. - Review failed tasks in the last 24 hours; ensure owners are notified and issues are tracked. - Check queued vs running tasks; confirm pools are not saturated. - Scan for import errors; fix or revert broken DAGs promptly. - Confirm alerts fired and were acknowledged for any incidents. ### Weekly - Noise review: list top 3 alerts by count; tune thresholds or add conditions. - Capacity review: compare queued tasks and run durations week-over-week. - Dashboard hygiene: remove panels no one uses; add annotations for incidents. - Dependency audit: note any upcoming upgrades to Airflow, DB drivers, or Python that could affect monitoring. ### Monthly - Disaster recovery drill: simulate scheduler outage and practice restore. - Update runbooks: keep step-by-step recovery up to date with current topology. - Expand coverage: add 1-2 more DAGs to alert scope based on business impact. ## Practical Examples and Expected Results ### 1. Email alert on failure - Trigger a failure in pilot_monitoring_example.flaky . - Expected: an email arrives within 1 minute. The message contains DAG ID, task ID, and run ID. ### 2. Webhook notification - For the same failure, a message posts to your channel. - Expected: a short text message with DAG and task identifiers. ### 3. Scheduler health change - Stop the scheduler process for 3 minutes in a controlled test window. - Expected: the health check reports unhealthy; your alert rule triggers; the dashboard shows flatlined heartbeat increments. - Rollback: restart scheduler; confirm the alert clears and heartbeat resumes. ### 4. SLA miss - Temporarily add time.sleep(1800) to ok_task to exceed the 20-minute SLA (for test only). - Expected: SLA miss notification via the same callback path; consider keeping SLA alerts at notification-only severity initially. ## Conclusion Start small and make it observable. A focused pilot with one DAG, a handful of high-signal metrics, and two or three clear alerts validates your monitoring approach quickly. Prove that alerts reach the right people, dashboards answer first-order questions, and logs provide the necessary detail. Once verified, expand coverage to your top DAGs, tune thresholds to reduce noise, and formalize incident response. This incremental approach keeps risk low, enables fast feedback, and builds a dependable monitoring layer for Apache Airflow as your orchestration needs grow.