Apache Airflow monitoring and alerts with practical examples: practical implementation guide

Apache Airflow orchestrates critical data and automation workflows. When schedules drift, retries escalate, or the scheduler stalls, you want to know quickly, with enough context to act. This guide delivers a practical, safe approach to monitoring and alerts for Airflow. You will:

• Identify the first metrics and log signals that matter.
• Configure baseline alerts (SLA misses, failures, scheduler health).
• Add a compact dashboard for situational awareness.
• Verify the setup with controlled tests.
• Prepare failure-mode playbooks and rollbacks.

The result is a monitoring foundation you can expand confidently.

Version and Environment Inventory

Before changes, write down a concise inventory so you know exactly where to configure and how to roll back. Use this template and replace values for your environment.

Inventory template (example fields, fill with your values):

• Airflow version: 2.6.x or later recommended
• Python version: 3.8+ recommended
• Executor: LocalExecutor or CeleryExecutor
• Webserver host: host or VIP, port 8080 (adjust as used)
• Scheduler host(s): hostnames
• Workers: count and hostnames (if using CeleryExecutor)
• Metadata DB: engine (e.g., Postgres), host, connection pooling
• Message broker: e.g., Redis or RabbitMQ (if CeleryExecutor)
• Log storage: local filesystem or remote (e.g., S3 or GCS)
• Email SMTP: hostname, from address
• Alert channels: email list, Slack webhook URL (if used)
• Config file path(s): airflow.cfg and any environment-variable overrides
• Access: shell account for Airflow service user and file backup location

Pre-requisites:

• Administrative access to update airflow.cfg or environment variables.
• Access to Airflow web UI and API (for /health checks).
• A notification channel (email SMTP or a webhook) you can safely test.
• Optionally: a metrics sink (e.g., StatsD-compatible endpoint or Prometheus collector) and a dashboard tool.

Safe Configuration Path

Scope the initial rollout to a small, verifiable pilot. Use one representative DAG to prove the loop end-to-end before scaling out.

Pilot scope

• One DAG with 3-5 tasks that runs at least hourly so you can iterate quickly.
• Baseline signals: scheduler health, DAG run duration, task failures, and queue/backlog.
• Two alert rules to start: scheduler unhealthy, and a spike in task failures for the pilot DAG.

1. Enable baseline email and metrics in Airflow

Add or confirm these airflow.cfg entries (create a dated backup first):

# airflow.cfg excerpts

[email]
email_backend = airflow.utils.email.send_email_smtp
smtp_host = smtp.example.com
smtp_starttls = True
smtp_ssl = False
smtp_user = [email protected]
smtp_password = YOUR_APP_PASSWORD
smtp_mail_from = [email protected]

[metrics]
# Enable StatsD if you have a StatsD-compatible sink.
statsd_on = True
statsd_host = 127.0.0.1
statsd_port = 8125
statsd_prefix = airflow
# Emit hostname in tags if supported by your metrics sink.

[logging]
# Ensure remote or local logs are accessible and retained.
base_log_folder = /var/log/airflow

[core]
# Keep a modest parallelism to avoid noisy spikes during pilot.
parallelism = 16

Environment variable equivalents (if you prefer env-based configuration):

AIRFLOW__EMAIL__EMAIL_BACKEND=airflow.utils.email.send_email_smtp
AIRFLOW__EMAIL__SMTP_HOST=smtp.example.com
AIRFLOW__EMAIL__SMTP_STARTTLS=True
AIRFLOW__EMAIL__SMTP_SSL=False
[email protected]
AIRFLOW__EMAIL__SMTP_PASSWORD=YOUR_APP_PASSWORD
[email protected]

AIRFLOW__METRICS__STATSD_ON=True
AIRFLOW__METRICS__STATSD_HOST=127.0.0.1
AIRFLOW__METRICS__STATSD_PORT=8125
AIRFLOW__METRICS__STATSD_PREFIX=airflow

Restart Airflow services in a maintenance window appropriate for your environment.

1. Add a failure callback and SLA to a pilot DAG

This constructed example shows a DAG-level on_failure_callback that posts to a webhook (e.g., a Slack Incoming Webhook). Adjust the URL and message fields to your channel. The code is intentionally simple for pilot purposes.

# dags/pilot_monitoring_example.py
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
from datetime import timedelta
import json
import urllib.request

WEBHOOK_URL = "https://hooks.example.com/services/T000/B000/XXXXX"  # replace

def notify(context):
    dag_id = context.get("dag_run").dag_id if context.get("dag_run") else context.get("dag").dag_id
    task_id = context.get("task_instance").task_id
    run_id = context.get("dag_run").run_id if context.get("dag_run") else "manual__test"
    text = f"Airflow alert: {dag_id}.{task_id} failed on run {run_id}"
    data = json.dumps({"text": text}).encode("utf-8")
    req = urllib.request.Request(WEBHOOK_URL, data=data, headers={"Content-Type": "application/json"})
    try:
        urllib.request.urlopen(req, timeout=5)
    except Exception:
        pass  # Do not fail the task due to notification issues

with DAG(
    dag_id="pilot_monitoring_example",
    start_date=days_ago(1),
    schedule_interval="@hourly",
    catchup=False,
    sla_miss_callback=notify,  # alert on SLA misses
    default_args={
        "email_on_failure": True,
        "email_on_retry": False,
        "email": ["[email protected]"],
        "on_failure_callback": notify,
        "retries": 1,
        "retry_delay": timedelta(minutes=5),
        "sla": timedelta(minutes=20),
    },
    tags=["monitoring-pilot"],
) as dag:

    def ok_task():
        return "ok"

    def flaky_task():
        raise RuntimeError("constructed failure for pilot test")

    t1 = PythonOperator(task_id="ok", python_callable=ok_task)
    t2 = PythonOperator(task_id="flaky", python_callable=flaky_task)
    t1 >> t2

Notes:

• The webhook and email are both used, so you can compare alert timeliness and reliability.
• SLA on default_args means each task has a 20-minute SLA; adjust to your DAG characteristics.

1. Pipe metrics to a collector

If your collector supports StatsD, point it to the host: port above. For Prometheus via statsd_exporter, an example mapping file might look like this (constructed example):

mappings:
  - match: "airflow.scheduler.heartbeat"
    name: "airflow_scheduler_heartbeat_count"
    labels: {}
  - match: "airflow.dag_processing.import_errors"
    name: "airflow_dag_import_errors_total"
  - match: "airflow.operator_successes"
    name: "airflow_task_success_total"
    labels:
      operator: "${1}"
  - match: "airflow.operator_failures"
    name: "airflow_task_fail_total"
    labels:
      operator: "${1}"

1. Create a compact dashboard

Start with 4-6 panels you can inspect at a glance:

• Scheduler heartbeat count and age trend (5m window).
• DAG run duration p50/p95 for the pilot DAG.
• Task success vs failure count (rate) for the pilot DAG.
• Queued tasks and running tasks (platform-wide).
• Import errors count (parsing problems).
• Optional: database connection pool saturation if exposed by your stack.

1. Define two or three targeted alert rules

Create alerts that are easy to reason about. Constructed examples below. Replace metric names and labels to match your collector.

1. Document owners and severity

• Pilot DAG owner: team or person who can fix failures.
• On-call rotation: who pages, when, and escalation.
• Severity mapping: for example, scheduler down = SEV-1; single DAG failures = SEV-3.

What to monitor first

This constructed table prioritizes early, high-signal metrics. Tune thresholds after a week of observations.

Verification and Diagnostics

Prove the setup with controlled tests and observable outcomes.

1. Health check the webserver and scheduler

Airflow exposes a basic health endpoint in recent versions. Example:

curl -s http://<webserver_host>:8080/health | jq .

Expected (constructed example):

{
  "metadatabase": {"status": "healthy"},
  "scheduler": {"status": "healthy"}
}

If you cannot access the endpoint, use server logs or service supervision tools to confirm processes are running.

1. Induce a controlled failure

• Unpause the pilot DAG in the UI.
• Manually trigger a run (Run -> Trigger DAG) or wait for the schedule.
• The task flaky fails by design. Within a minute, you should see:
• An email sent to [email protected].
• A webhook notification in your channel.
• A bump in task failure metrics in your dashboard.

1. Verify metrics emission

If using statsd_exporter on localhost port 9102 (constructed example):

curl -s http://127.0.0.1:9102/metrics | grep -E "airflow_"

Expected: several airflow_* series including scheduler, task successes/failures, and import errors.

1. Inspect logs for signal quality

Open the flaky task log in the UI. Confirm the exception is visible and timestamps align with your alert timestamps. Ensure logs are retained where your operators can read them.

1. Validate dashboard panels

• Scheduler heartbeat panel: should show regular increments.
• DAG run duration p95: should reflect recent runs; the failed run may be shorter but should appear on the timeline.
• Task success/failure: visible spike at test time.
• Queued tasks: remain near normal if capacity is adequate.

1. Diagnose if something did not work

• No email: verify SMTP host, credentials, and firewall rules. Try a test email via a simple Python script using the same SMTP settings.
• No webhook alert: test the webhook URL with curl; check proxy rules.
• No metrics: confirm statsd_on is true, exporter is running, and ports are open. Look for errors in scheduler logs about metrics clients.
• Health check fails: restart affected Airflow services in your maintenance procedure and check the metadata DB connectivity.

Failure Modes and Recovery

Common issues you may encounter and how to recover safely.

Rollback playbook

• Back up airflow.cfg before changes; to roll back, restore the previous copy and restart services.
• If a callback causes failures, remove the on_failure_callback and sla_miss_callback lines from the pilot DAG, redeploy, and clear only the affected DAG runs.
• Disable noisy alert rules by toggling them off in your alerting tool, then revisit thresholds.
• If the metrics path causes instability, set statsd_on = False and restart to confirm stability.
• Always re-run the /health check and a manual pilot DAG run to validate recovery.

Incident response workflow (constructed example)

• Triage within 5 minutes: classify severity; if scheduler unhealthy, declare SEV-1 and page on-call.
• Contain: pause high-churn DAGs if needed to reduce load during recovery.
• Remediate: restart components as necessary; apply targeted fixes (e.g., revert a faulty DAG file).
• Verify: run /health, confirm dashboard normalization, and check new runs of affected DAGs.
• Document: note cause, fix, and threshold adjustments for the next noise review.

Operations Checklist

Daily

• Check Airflow /health endpoint for green status on metadatabase and scheduler.
• Review failed tasks in the last 24 hours; ensure owners are notified and issues are tracked.
• Check queued vs running tasks; confirm pools are not saturated.
• Scan for import errors; fix or revert broken DAGs promptly.
• Confirm alerts fired and were acknowledged for any incidents.

Weekly

• Noise review: list top 3 alerts by count; tune thresholds or add conditions.
• Capacity review: compare queued tasks and run durations week-over-week.
• Dashboard hygiene: remove panels no one uses; add annotations for incidents.
• Dependency audit: note any upcoming upgrades to Airflow, DB drivers, or Python that could affect monitoring.

Monthly

• Disaster recovery drill: simulate scheduler outage and practice restore.
• Update runbooks: keep step-by-step recovery up to date with current topology.
• Expand coverage: add 1-2 more DAGs to alert scope based on business impact.

Practical examples and expected results

1. Email alert on failure

• Trigger a failure in pilot_monitoring_example.flaky.
• Expected: an email arrives within 1 minute. The message contains DAG ID, task ID, and run ID.

1. Webhook notification

• For the same failure, a message posts to your channel.
• Expected: a short text message with DAG and task identifiers.

1. Scheduler health change

• Stop the scheduler process for 3 minutes in a controlled test window.
• Expected: the health check reports unhealthy; your alert rule triggers; the dashboard shows flatlined heartbeat increments.
• Rollback: restart scheduler; confirm the alert clears and heartbeat resumes.

1. SLA miss

• Temporarily add time.sleep(1800) to the ok_task to exceed the 20-minute SLA (constructed for test).
• Expected: SLA miss notification via the same callback path; consider keeping SLA alerts at notification-only severity initially.

Conclusion

Start small and make it observable. A focused pilot with one DAG, a handful of high-signal metrics, and two or three clear alerts validates your monitoring approach quickly. Prove that alerts reach the right people, dashboards answer first-order questions, and logs provide the necessary detail. Once verified, expand coverage to your top DAGs, tune thresholds to reduce noise, and formalize incident response.

This incremental approach keeps risk low, enables fast feedback, and builds a dependable monitoring layer for Apache Airflow as your orchestration needs grow.