Reliable data orchestration depends on repeatable, safe deployments. Apache Airflow CI/CD turns changes to DAGs and plugins into shippable increments with fast feedback and minimal risk. In this guide you will: - Inventory versions and topology so results are predictable. - Set up a small but realistic CI workflow to lint, parse, and test DAGs. - Deploy with an atomic, symlink-based release mechanism that makes rollback instant. - Verify expected behavior using Airflow CLI. - Diagnose common failure modes and recover safely. A structured approach helps teams move from proposed changes to reviewed, shippable increments. Separating authoring, validation, and deployment reduces rework and speeds learning in the early stages of automation. ## Version and Environment Inventory Consistency starts with locking versions, listing executors, and deciding how Airflow discovers your DAGs. The following version matrix is a constructed example you can adapt to your stack.
ComponentVersionPurposeNotes
Apache Airflow2.7.3OrchestratorUpgrade together across environments
Python3.10RuntimeMatch local and CI interpreters
ExecutorLocal/CelerySchedulingChoose per scale; examples are executor-agnostic
Metadata DBPostgres 14State storeEnsure same minor version in staging/prod
OSUbuntu 22.04 LTSHostsKeep glibc/openSSL parity across environments
Prerequisites: - Git repository containing: - dags/ for DAGs - plugins/ for custom operators/hooks - tests/ with DAG import and policy tests - requirements.txt and constraints.txt to lock dependencies - scripts/ with parse, deploy, and rollback helpers - Python 3.9+ available in local and CI runners - Airflow CLI access on staging and production hosts - SSH access or a pull-from-git mechanism on Airflow hosts - Documented Airflow Connections and Variables for each environment Topology notes: - Point Airflow's dags_folder to a stable path (for example, /opt/airflow/dags/current ) and manage a releases/ directory with symlinks. - Ensure scheduler and webserver users can read from the DAGs folder and all subdirectories. ## Safe Configuration Path Start small. The first pilot should be narrow, measurable, and easy to inspect locally before deployment. Use a smoke DAG and a few tests to prove the path end to end. ### Repository Layout repo-root/ ├── dags/ │ └── example_smoke_dag.py ├── plugins/ ├── tests/ │ ├── test_dag_imports.py │ └── test_dag_policies.py ├── requirements.txt ├── constraints.txt └── scripts/ ├── deploy_push.sh └── rollback.sh ### Example Smoke DAG # file: dags/example_smoke_dag.py from airflow import DAG from airflow.operators.bash import BashOperator from datetime import datetime with DAG( dag_id="example_smoke_dag", start_date=datetime(2024, 1, 1), schedule=None, catchup=False, tags=["smoke"], ) as dag: echo_env = BashOperator( task_id="echo_env", bash_command="echo AIRFLOW_VERSION=$(airflow version)" ) ### Pin Dependencies for Repeatability # requirements.txt apache-airflow==2.7.3 pendulum==2.1.2 # constraints.txt apache-airflow==2.7.3 pendulum==2.1.2 ### Lightweight Tests Without a Scheduler DAG import test: # file: tests/test_dag_imports.py import os from airflow.models import DagBag def test_no_import_errors(): dag_folder = os.path.join(os.getcwd(), "dags") dag_bag = DagBag(dag_folder=dag_folder, include_examples=False) assert len(dag_bag.import_errors) == 0, f"DAG import failures: {dag_bag.import_errors}" assert len(dag_bag.dags) > 0, "No DAGs were discovered" Policy test to prevent runaway backfills by forbidding dynamic start_date : # file: tests/test_dag_policies.py import glob def test_no_dynamic_start_date(): for path in glob.glob("dags/**/*.py", recursive=True): with open(path, "r", encoding="utf-8") as f: src = f.read() assert "datetime.now(" not in src, f"Dynamic start_date in {path}" assert "pendulum.now(" not in src, f"Dynamic start_date in {path}" ### Local Test Run python -m pip install -r requirements.txt -c constraints.txt pytest -q # Expected: tests pass, zero import errors ### Deployment Options - Pull-based: Airflow hosts pull from a Git branch or release tag to the releases/ directory and update the current symlink. This minimizes CI permissions on production. - Push-based: CI connects over SSH to rsync DAGs/plugins to a new release directory, then atomically flips the current symlink. Configure dags_folder in airflow.cfg on each host: [core] dags_folder = /opt/airflow/dags/current ### Push-Based Deploy Script # file: scripts/deploy_push.sh #!/usr/bin/env bash set -euo pipefail AIRFLOW_HOST="${AIRFLOW_HOST:?set AIRFLOW_HOST}" AIRFLOW_DAGS_ROOT="/opt/airflow/dags" RELEASES_DIR="${AIRFLOW_DAGS_ROOT}/releases" RELEASE_ID="${1:?usage: $0 }" SRC_DIR="${2:-$(pwd)}" ssh "$AIRFLOW_HOST" "mkdir -p ${RELEASES_DIR}/${RELEASE_ID}" rsync -av --delete "${SRC_DIR}/dags/" "$AIRFLOW_HOST:${RELEASES_DIR}/${RELEASE_ID}/" rsync -av --delete "${SRC_DIR}/plugins/" "$AIRFLOW_HOST:${RELEASES_DIR}/${RELEASE_ID}/plugins/" || true ssh "$AIRFLOW_HOST" "ln -sfn ${RELEASES_DIR}/${RELEASE_ID} ${AIRFLOW_DAGS_ROOT}/current && ls -l ${AIRFLOW_DAGS_ROOT}" ### Rollback Script # file: scripts/rollback.sh #!/usr/bin/env bash set -euo pipefail AIRFLOW_HOST="${AIRFLOW_HOST:?set AIRFLOW_HOST}" AIRFLOW_DAGS_ROOT="/opt/airflow/dags" PREV_RELEASE_ID="${1:?usage: $0 }" ssh "$AIRFLOW_HOST" "ln -sfn ${AIRFLOW_DAGS_ROOT}/releases/${PREV_RELEASE_ID} ${AIRFLOW_DAGS_ROOT}/current && ls -l ${AIRFLOW_DAGS_ROOT}" Note: Some environments detect symlink swaps immediately; others require a touch to trigger file change detection. If the scheduler does not see new DAGs within a minute, gracefully restart the scheduler or touch a file inside the DAG directory. ## Verification and Diagnostics ### Pre-Deploy Checks (Local or CI) - Install dependencies with constraints - Lint and unit tests - Parse DAGs with DagBag to ensure zero import errors - Optional: dry-run a task with tasks test ### Example Commands and Expected Outputs python -m pip install -r requirements.txt -c constraints.txt pytest -q # Expected: "2 passed" (or more), and no import_errors # Optional task test without creating a DAG run airflow tasks test example_smoke_dag echo_env 2024-01-01 # Expected last line: "Task exited with return code 0" or state SUCCESS ### Post-Deploy Checks on Staging or Production # DAG discovery airflow dags list | grep example_smoke_dag # Expected: one line showing example_smoke_dag # Task listing airflow tasks list example_smoke_dag # Expected: echo_env listed # Smoke trigger (manual) RUN_ID="ci-smoke-$(date +%s)" airflow dags trigger -r "$RUN_ID" example_smoke_dag sleep 5 # Observe runs airflow dags list-runs -d example_smoke_dag | head -n 5 # Expected: most recent run with state running/success # Inspect task instance log (adjust execution date if needed) # For manual runs, use the UI or list-runs to find execution date. ### What to Observe - The DAG appears in the list within 60 seconds of deploy. - The smoke task succeeds and logs include an Airflow version echo. - Scheduler logs show zero import errors and no permission errors while scanning the DAG directory. Tip: Attach a simple environment sensor to your smoke DAG to fail quickly if required Connections or Variables are missing. For example, a PythonOperator that checks for an expected connection ID and raises an exception if not found. ## Failure Modes and Recovery Use the table below to connect symptoms with likely causes, diagnostics, and safe fixes. Items are constructed examples you can adapt.
SymptomLikely CauseQuick DiagnosticSafe FixRollback Approach
DAG missing after deploydags_folder not pointing to current, or permission issueairflow config get-value core dags_folder; ls -l /opt/airflow/dagsPoint to /opt/airflow/dags/current; ensure r-x perms for scheduler userFlip symlink back to previous release
ImportError in scheduler logsMissing or incompatible Python dependencygrep -i importerror $AIRFLOW_HOME/logs/scheduler/* -nPin version in constraints.txt; add to requirements.txt and redeployRevert symlink; reinstall previous requirements if they changed
Runaway backfill on first deployDynamic or far-past start_date with catchup enabledInspect DAG code; airflow dags list-runs -d <dag>Set catchup=False and use backfill intentionally with a bound windowRevert DAG to prior version; clear unwanted runs
Task fails only in prodMissing Connection/Variable in prodairflow connections get <id>; airflow variables get <name>Create the same IDs in prod; avoid reading secrets directly in DAG codeRoll back DAG; add environment guard in smoke checks
Scheduler sees stale codeSymlink swap not detected; process watching old inodels -l /opt/airflow/dags/current; check mtime updatesTouch a file or restart scheduler gracefullyRoll back symlink; restart to restore prior state
### Recovery Patterns - Atomic releases: Keep N historical releases in /opt/airflow/dags/releases/ . The current symlink points at the active one. - Instant rollback: Repoint the current symlink to the previous release. No file copying is necessary. - Dependency isolation: If you install Python packages system-wide, rollback includes restoring prior constraints. Prefer a virtualenv per release when feasible. - Clear unintended runs: If a DAG created excess backfills, disable it, set catchup=False , and clear any unwanted task instances after rollback. ## Operations Checklist ### Pre-Merge - Confirm DAG adheres to team policies (static start_date , bounded retries, proper tags) - Add or update unit tests for custom operators and DAG parsing - Pass import and policy tests locally ### Pre-Deploy (CI) - Install with constraints: pip install -r requirements.txt -c constraints.txt - Run pytest -q and ensure zero import errors - Optional: airflow tasks test for critical tasks - Produce a release identifier (timestamp or commit SHA) ### Deploy - Create release directory on target host: /opt/airflow/dags/releases/ - Sync dags/ and plugins/ to the release directory - Atomically ln -sfn the current symlink to the new release ### Post-Deploy Validation - airflow dags list | grep shows the DAG - airflow tasks list lists expected tasks - Trigger smoke DAG and verify success within SLA - Scan scheduler logs for import or permission errors ### Rollback (If Any Validation Fails) - Repoint current to previous release: rollback.sh - Confirm DAG list shows previous versions - Re-run smoke checks to confirm health - Open an issue with diagnostics and a minimal reproduction ### Review and Continuous Improvement - Capture mean time to detect (MTTD) and to recover (MTTR) - Add policy tests for any new class of incident you encountered - Expand smoke DAGs to include critical Connections and Variables checks ## Practical Examples in Context ### End-to-End Flow for a Small Change - Developer edits example_smoke_dag.py to add a second Bash task that prints the hostname. - Runs locally: - pip install -r requirements.txt -c constraints.txt - pytest -q → all tests pass - airflow tasks test example_smoke_dag echo_env 2024-01-01 → SUCCESS - Opens a change request; CI runs the same tests and produces a release ID like 20240115-abcdef . - Upon approval, CI executes deploy_push.sh 20240115-abcdef to staging. - On staging host: - airflow dags list | grep example_smoke_dag → visible - airflow tasks list example_smoke_dag → shows echo_env and new task - Trigger and observe logs → both tasks succeed - Promote to production using the same release ID; validate; monitor for 30 minutes. - If any issue arises, run rollback.sh 20240110-123456 to instantly revert. ## Conclusion You now have a practical, defensible approach to Apache Airflow CI/CD. By inventorying versions and topology, you ensure changes behave the same across environments. Fast, deterministic tests catch import errors and policy violations before they reach production. Atomic deployments with a releases directory and a current symlink give you instant rollback without file copying. Verification with Airflow CLI and a simple smoke DAG confirms environment health end to end. Finally, a clear map of common failure modes—missing DAGs, import errors, runaway backfills, environment-specific task failures, and stale scheduler state—lets you diagnose and recover quickly. A narrow, inspectable pilot builds trust fast and creates a pattern you can scale across your entire orchestration estate.