Apache Airflow Architecture Explained with Practical Examples

Intro

Apache Airflow is the leading open-source platform for authoring, scheduling, and monitoring data pipelines. At its core, Airflow lets you define workflows as code, making them versionable, testable, and maintainable. Understanding its architecture is essential for developers and operations teams who need to build reliable data orchestration systems. This guide explains Airflow's key components, how data and control flow through the system, and practical considerations for running it in production.

We will start with a version and environment inventory to set a baseline, then walk through safe configuration paths, verification techniques, failure modes and recovery, and finally provide an operations checklist. Along the way, you will see concrete examples, commands, and configuration snippets that illustrate each concept.

Version and Environment Inventory

Before deploying or operating Airflow, you should know exactly which version you are running, what components are involved, and the environment topology. Airflow releases can vary significantly in behavior, so pinning the version is critical.

Airflow Version

Airflow follows semantic versioning. As of this writing, the latest stable release is 2.8.1. Always check the official documentation for the version you use. To find the installed version, run:

airflow version

Example output:

2.8.1

If you are using a Python environment, you can also check with:

pip show apache-airflow | grep Version

Output:

Version: 2.8.1

Core Components Inventory

Airflow consists of several core components that may run on a single machine for development or across many machines for production. The main components are:

• Scheduler: Triggers scheduled workflows and submits tasks to the executor.
• Executor: Handles running tasks. Common executors include SequentialExecutor, LocalExecutor, CeleryExecutor, and KubernetesExecutor.
• Webserver: Provides the user interface for monitoring and managing DAGs and tasks.
• Metadata Database: Stores state and metadata. Typically PostgreSQL or MySQL.
• DAG Directory: A folder containing DAG definition files, usually written in Python.

In a typical production setup, these components are distributed. For example, the scheduler and webserver may run on separate machines, with Celery workers executing tasks. For this guide, we assume a fresh installation on a single Ubuntu 22.04 LTS server with Python 3.10 and PostgreSQL 14 as the metadata database. The topology is:

Prerequisites

Before installing Airflow, ensure your environment meets these prerequisites:

• Python 3.8, 3.9, 3.10, or 3.11
• pip 20.2.4+
• A database such as PostgreSQL 11+, MySQL 8+, or SQLite for development only
• Enough disk space for logs and DAGs
• System dependencies: libpq-dev for PostgreSQL, libmysqlclient-dev for MySQL

Installing Airflow

Airflow should be installed in a dedicated Python virtual environment. Use the following commands to install Airflow 2.8.1 with PostgreSQL support:

python3 -m venv airflow_env
source airflow_env/bin/activate
pip install "apache-airflow[postgres]==2.8.1"

After installation, initialize the metadata database:

export AIRFLOW_HOME=~/airflow
airflow db init

This creates the necessary tables in the configured database. You can then start the webserver and scheduler for a development setup:

airflow webserver -p 8080
airflow scheduler

But this is just the beginning. In the next sections, we'll configure Airflow for a production-like environment and explain the architecture in depth.

Safe Configuration Path

Airflow's behavior is controlled by configuration options defined in airflow.cfg, environment variables, or command-line arguments. A safe configuration path means making changes in a controlled, reversible way. We'll focus on key settings that affect architecture and performance, and provide a concrete example of configuring a Celery executor with PostgreSQL and Redis.

Configuration Hierarchy

Airflow reads configuration in this order:

• airflow.cfg file (default in $AIRFLOW_HOME)
• Environment variables with prefix AIRFLOW__ (e.g., AIRFLOW__CORE__SQL_ALCHEMY_CONN)
• Command-line options (highest precedence)

For production, using environment variables is common to avoid storing secrets in files. However, for this guide, we'll edit the airflow.cfg file directly for clarity.

Key Configuration Sections

• [core]: General settings like dags_folder, executor.
• [database]: SQLAlchemy connection string.
• [celery]: Celery-specific settings.
• [scheduler]: Scheduler behavior.
• [webserver]: Web UI settings.

Example: Configure Celery Executor with PostgreSQL and Redis

Suppose we want to set up a distributed architecture using CeleryExecutor for parallel task execution across multiple worker machines. We'll need a PostgreSQL database for metadata and Redis as the Celery broker.

1. PostgreSQL Setup

Create a database and user for Airflow:

CREATE DATABASE airflow;
CREATE USER airflow_user WITH PASSWORD 'strongpassword';
GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow_user;

2. Airflow Configuration

Edit airflow.cfg:

[core]
executor = CeleryExecutor
dags_folder = /opt/airflow/dags

[database]
sql_alchemy_conn = postgresql+psycopg2://airflow_user:strongpassword@postgres-01/airflow

[celery]
broker_url = redis://redis-01:6379/0
result_backend = db+postgresql://airflow_user:strongpassword@postgres-01/airflow

3. Initialize Database

Run:

airflow db init

Expected output:

DB: postgresql+psycopg2://airflow_user:***@postgres-01/airflow
[2024-02-20 12:34:56,789] {db.py:1486} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
...
Done.

4. Create Airflow User

Create an admin user for the webserver login:

airflow users create \
    --username admin \
    --firstname Admin \
    --lastname User \
    --role Admin \
    --email [email protected]

5. Start Services

Start the scheduler, webserver, and a Celery worker:

airflow scheduler
airflow webserver -p 8080
airflow celery worker

Each command runs in a separate terminal or as a service. The expected output includes logs showing the scheduler parsing DAGs, the webserver starting, and the worker registering with the broker.

Scoped Implementation Choices

When making configuration changes, scope them to a single component or feature at a time. For example, if you change the executor, test it with a simple DAG before rolling out more complex workflows. Keep the old configuration as a fallback. Use version control for all configuration files and DAGs.

Understanding DAGs and Operators

A DAG (Directed Acyclic Graph) defines the workflow. It is a Python script that creates a DAG object and defines tasks using operators. Operators determine what a task does. For example, BashOperator runs a shell command, PythonOperator calls a Python function, and EmailOperator sends an email.

Example DAG file example_hello_world.py:

from datetime import datetime
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator

def print_hello():
    print("Hello from Python!")
    return "Hello returned value"

with DAG(
    'hello_world',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily',
    catchup=False,
) as dag:
    bash_task = BashOperator(
        task_id='bash_task',
        bash_command='echo "Hello from Bash!"',
    )
    python_task = PythonOperator(
        task_id='python_task',
        python_callable=print_hello,
    )
    bash_task >> python_task  # sets dependency

This DAG has two tasks: a bash task and a Python task, with the Python task depending on the bash task. The schedule_interval is set to daily.

Scheduling and Data Flow

The scheduler is the heart of Airflow. It continuously checks for DAGs, triggers tasks according to schedule and dependencies, and submits them to the executor. The data flow in Airflow can be summarized as:

1. The scheduler reads DAG files from the DAG folder and parses them.
2. The scheduler determines which tasks are ready to run based on schedule, dependencies, and state in the metadata database.
3. The scheduler submits task instances to the executor.
4. The executor runs the task, either locally or on a worker, and updates the task status in the metadata database.
5. The webserver reads from the metadata database to display task states and logs.

The control flow (scheduler decisions) is separate from the data flow (task execution). This separation allows scaling the executor independently from the scheduler.

Verification and Diagnostics

After configuring Airflow, you need to verify that it is working correctly. This section covers observable checks and expected results.

Verify Scheduler and Webserver

• Check if the scheduler process is running:

ps aux | grep 'airflow scheduler'

Expected output includes the scheduler process.

• Access the webserver UI at http://airflow-web-01:8080. You should see the Airflow home page.

Verify DAG Parsing

The scheduler parses DAGs periodically. If there is an error in a DAG file, it will appear in the UI with a red error message. You can also manually test a DAG file by running:

python /path/to/dag_file.py

If no error occurs, the DAG file is syntactically correct. However, note that this does not guarantee that tasks will run correctly.

Verify Database Connectivity

Check that the scheduler can connect to the metadata database:

airflow db check

Expected output:

Connection: OK

If the connection fails, you will see an error indicating the problem, such as authentication failure or database not running.

Verify Executor

For CeleryExecutor, check the Celery worker status:

airflow celery inspect active

This command queries the worker for active tasks. If the worker is up, it will return an empty list when no tasks are running:

[]

You can also check the broker connection by running:

airflow celery flower

Flower is a web-based tool for monitoring Celery. It will start on port 5555 by default.

Run a Test DAG

Create a simple DAG that prints a message, then trigger it manually and observe the task logs.

1. Add the DAG file to the DAG folder.
2. In the UI, unpause the DAG (toggle the switch).
3. Click the "Trigger DAG" button.
4. Go to the Graph View and watch the task turn green, indicating success.
5. Click on the task instance and open the logs. You should see the output from the task.

Example log output for the bash task:

[2024-02-20 13:00:00,123] {bash.py:135} INFO - Running command: echo "Hello from Bash!"
[2024-02-20 13:00:00,456] {bash.py:142} INFO - Output:
[2024-02-20 13:00:00,789] {bash.py:146} INFO - Hello from Bash!
[2024-02-20 13:00:01,012] {bash.py:150} INFO - Command exited with return code 0

Monitoring Metrics

Airflow exposes metrics in StatsD format. You can configure statsd_on = True in airflow.cfg and set statsd_host and statsd_port. Then use a tool like Prometheus with StatsD exporter to collect and visualize metrics. Key metrics include:

• scheduler.tasks.running
• scheduler.tasks.starving
• dag_processing.import_errors
• executor.open_slots

Regularly monitoring these metrics helps ensure the health of your Airflow deployment.

Failure Modes and Recovery

Even with careful configuration, Airflow can fail. Understanding common failure modes and how to recover is crucial.

Common Failure Points

Recovery Steps

Scheduler Failure

If the scheduler crashes, check the logs located in $AIRFLOW_HOME/logs/scheduler/. Look for traceback or out-of-memory errors. Restart the scheduler:

airflow scheduler

If it crashes repeatedly, investigate the cause. Common issues include database connection problems or a bug in a DAG that causes excessive resource consumption.

Metadata Database Outage

If the database is down, restore it from backup if necessary. Ensure the database service is running and accessible. Check the sql_alchemy_conn string in airflow.cfg for errors.

Executor Worker Down

For Celery, if a worker goes down, tasks will remain in the queue. Start a new worker:

airflow celery worker

You may need to restart the worker if it consumes too much memory. Monitor workers with Flower.

Broker Outage

If Redis goes down, the scheduler and workers cannot communicate. Restart Redis and ensure the connection string is correct. Consider using Redis with persistence enabled to avoid data loss.

DAG Parse Error

Fix the syntax error in the DAG file. The scheduler will automatically pick up the corrected file after a short interval (default min_file_process_interval is 30 seconds).

Rollback and Recovery Strategies

• DAG Code Rollback: If a new DAG version causes problems, revert the DAG file to the previous version using version control. The scheduler will parse the old code. Use catchup=False to avoid backfilling unintentionally.
• Backfill with Caution: To re-run historical dates, use the airflow dags backfill command. For example:

airflow dags backfill -s 2024-02-01 -e 2024-02-10 hello_world

This will run the DAG for each day from Feb 1 to Feb 10. Be careful when using backfill in production because it can trigger many tasks and overload resources.

• Clear Task States: If a task fails, you can clear its state to re-run it. In the UI, click on the failed task and select "Clear". Alternatively, using CLI:

airflow tasks clear -s 2024-02-20 -e 2024-02-20 hello_world bash_task

This clears the task instance state for that date, causing it to be re-run.

Recovery Checks

After any recovery, verify:

• The scheduler is heartbeating (check logs for "Heartbeat" messages).
• The webserver is accessible and shows DAGs.
• The executor is processing tasks (check for task starts in logs).
• The database connection is healthy (airflow db check).

Common Pitfalls and How to Avoid Them

Airflow pitfalls often stem from misunderstanding its architecture or misconfiguring components. Here are the most common mistakes we see, why they happen, and how to avoid or recover from them.

1. Using SQLite in Production

Why it happens: SQLite is the default metadata database and works fine for a quick local test, so teams sometimes leave it unchanged when moving to production.

How to avoid: Always configure a robust database like PostgreSQL or MySQL for any non-trivial deployment. SQLite does not support concurrent writes well and can cause scheduler lockups. Set the sql_alchemy_conn in [database] to a production-grade database from day one.

Recovery: If you discover SQLite in production, migrate to PostgreSQL/MySQL by exporting the metadata (using airflow db export) and importing it into the new database after changing the connection string. Then restart the scheduler and webserver.

2. Ignoring start_date and catchup

Why it happens: New users often set start_date to the current date and leave catchup=True (the default in Airflow 1.x) without realizing the scheduler will try to backfill every missed interval.

How to avoid: Set catchup=False on the DAG unless you explicitly want backfills. Choose a start_date that is static and in the past, not datetime.now(), to avoid timezone issues.

Recovery: If a DAG unexpectedly triggers many backfill tasks, set catchup=False and clear the unwanted task instances using the UI or CLI.

3. Overloading the Scheduler with Too Many DAGs or Short Intervals

Why it happens: Teams create many DAGs with schedule_interval of every minute or even seconds, not accounting for the scheduler's parsing and scheduling overhead.

How to avoid: Monitor scheduler.tasks.starving and DAG parsing times. Batch small tasks into larger DAGs where possible, and use sensible intervals (e.g., no less than 5 minutes for most use cases). Consider using a dedicated DAG processor if needed.

Recovery: If the scheduler is overwhelmed, reduce the number of DAGs or increase intervals. You can also adjust scheduler.max_tis_per_query and scheduler.parsing_processes based on your hardware.

4. Not Setting result_backend for Celery

Why it happens: CeleryExecutor requires a result backend to store task results. If not configured, tasks may run but the metadata database won't be updated correctly, leading to duplicate runs or stuck states.

How to avoid: Always set result_backend in [celery] to either a database (as shown earlier) or a message broker like Redis.

Recovery: Update airflow.cfg with the correct result_backend, restart the scheduler and workers, and clear any stuck task instances.

5. Storing Secrets in DAG Files or airflow.cfg

Why it happens: Developers often hardcode credentials in DAG code or configuration for convenience.

How to avoid: Use Airflow Connections and Variables stored in the metadata database (encrypted at rest if configured) or an external secrets backend like HashiCorp Vault or AWS Secrets Manager. Never put secrets in code.

Recovery: If secrets were exposed, rotate them immediately and move them to the secrets backend. Update DAGs to use Connection.get_connection_from_secrets() or environment variables.

6. No Monitoring or Alerting

Why it happens: Small teams may skip monitoring initially, relying on manual checks.

How to avoid: Set up StatsD metrics and integrate with Prometheus/Grafana. Configure alerts for key metrics like import_errors > 0, scheduler.tasks.starving > threshold, or worker down.

Recovery: Start by enabling StatsD and exporting to a monitoring system. Create dashboards and alert rules gradually.

Operations Checklist

Running Airflow in production requires ongoing operational discipline. Here is a repeatable checklist for daily, weekly, and monthly reviews. For each item, assign a single accountable owner (e.g., a specific engineer on your team) and set a review cadence.

Daily Checks

Perform these checks daily. Accountability: The on-call data engineer (e.g., Priya Shah, Data Engineer) reviews this list every morning.

• [ ] Scheduler process running and heartbeating (owner: Priya Shah, check logs for "Heartbeat" messages).
• [ ] Webserver responsive (owner: Priya Shah, curl or access UI).
• [ ] No import errors in DAGs (owner: Priya Shah, check UI dashboard or import_errors metric).
• [ ] No tasks stuck in queued state for too long (owner: Priya Shah, view task durations).
• [ ] No excessive task failures (owner: Priya Shah, monitor failure metrics).
• [ ] Database connection stable (owner: Priya Shah, run airflow db check).

Weekly Checks

Perform these checks every Monday morning. Accountability: The data platform lead (e.g., Marcus Chen, Data Platform Lead) owns this review.

• [ ] Review task logs for anomalies (owner: Marcus Chen).
• [ ] Monitor resource usage (CPU, memory) of scheduler, webserver, workers (owner: Marcus Chen).
• [ ] Check disk space for logs and DAG files; rotate logs if needed (owner: Marcus Chen).
• [ ] Verify backups of metadata database (owner: Marcus Chen).
• [ ] Review DAG code changes for potential issues (owner: Marcus Chen).

Monthly Checks

Perform these checks on the first business day of each month. Accountability: The data engineering manager (e.g., Sarah Johnson, Data Engineering Manager) owns this review.

• [ ] Apply security patches and update Airflow to a stable version after testing (owner: Sarah Johnson).
• [ ] Review user accounts and permissions (owner: Sarah Johnson).
• [ ] Test disaster recovery procedures (owner: Sarah Johnson).
• [ ] Review performance and consider scaling executor if needed (owner: Sarah Johnson).
• [ ] Validate that DAGs still align with business requirements (owner: Sarah Johnson).

Example of a Monitoring Command

Use the Airflow CLI to list DAGs and their schedules:

airflow dags list

Output:

DAG ID       | Schedule | Owner
-------------------------------------
hello_world  | @daily   | admin

This helps ensure that expected DAGs are present and correctly scheduled.

Conclusion

Understanding Apache Airflow's architecture is fundamental to building reliable data pipelines. We have covered the core components, data and control flow, safe configuration with a practical Celery setup, verification steps, failure modes and recovery, common pitfalls, and an operations checklist. By following the guidelines in this guide, you can deploy and maintain Airflow effectively.

As next steps, consider:

• Experiment with different executors to see how they affect scaling.
• Explore advanced features like XComs for passing data between tasks.
• Implement monitoring with Prometheus and Grafana for better visibility.
• Integrate with a secrets backend for managing credentials.
• Set up CI/CD for your DAGs to ensure code quality.

Airflow is a powerful tool, and with a solid architectural understanding, you can leverage it to orchestrate complex workflows with confidence.