## Intro Apache Airflow is a powerful workflow orchestration platform, but daily operations often hinge on knowing the right commands and using them safely. A basic command run at the wrong time or against the wrong component can turn a small issue into an outage. This guide focuses on practical commands that operators, DevOps engineers, and technical startup teams use to verify, diagnose, and safely manage Airflow deployments. You will learn how to check versions, assess health, inspect tasks, and recover from common failures, all with concrete examples and expected outputs. We emphasize operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths. By the end, you will have a reusable checklist for daily Airflow operations. ## Version and Environment Inventory Before running any command, establish the exact version and topology of your Airflow installation. Different versions have different commands and behaviors. For example, Airflow 3 introduced an API server and a separately running DAG processor, which changes how you inspect and manage the system. Start with: airflow version Expected output for a recent release: 2.10.3 If you are on Airflow 3, the output may look like 3.0.0 . Note the major version because many commands differ. Document also the executor, metadata database, and deployment method (e.g., Docker, Kubernetes, bare metal). A quick way to get configuration details: airflow config get-value core executor Expected output: CeleryExecutor or LocalExecutor , KubernetesExecutor , etc. This tells you how tasks are executed and what components are involved. For Airflow 3, verify the health of the API server and DAG processor separately. Use the health endpoint with curl: curl -s http://localhost:8080/api/v2/monitor/health The response is a JSON object; do not treat HTTP 200 alone as proof of health. Inspect fields like scheduler , triggerer , dag_processor , and metadata_db . A healthy scheduler may show: { "scheduler": { "status": "healthy", "latest_scheduler_heartbeat": "2025-03-25T12:00:00Z" }, "metadata_db": { "status": "healthy" } } If a component is unhealthy, investigate before making changes. For upgrades, especially from Airflow 2.7+ to 3.x, always review release notes and provider compatibility. Back up your metadata database before migration. Test representative DAGs on a staging environment. Then run: airflow db migrate Do not substitute a restart for a migration. After migration, verify with airflow db check and run a few DAGs manually. ## Safe Configuration Path Configuration changes can affect the entire cluster. Use a controlled process: observe, plan, change one item, verify, and document rollback. Start by viewing the current configuration without exposing secrets. Use: airflow config list This prints all options, but values may be redacted for sensitive items. To see a specific option: airflow config get-value core dags_folder Expected output: /opt/airflow/dags When changing a configuration, use the environment variable or airflow.cfg, but avoid editing live files directly if you use a configuration management system. For example, to increase the scheduler heartbeat interval in Airflow 2, you might set scheduler.heartbeat_sec = 5 in airflow.cfg. After the change, restart only the scheduler process (if running as a separate service), not the whole stack. Verify with: airflow jobs check --job-type SchedulerJob --hostname $(hostname) Expected output on success: Found one alive job. For Airflow 3, some configurations are applied via environment variables. Example: set AIRFLOW__CORE__DAGS_FOLDER=/new/path and restart the DAG processor. Verify the new path is picked up by checking the DAG processor logs or running airflow dags list . Always keep a record of the previous value and the reason for the change. If the change degrades performance, revert promptly. ## Verification and Diagnostics Daily operations require frequent verification that the system is functioning and DAGs are running as expected. Use read-only commands first. Check the scheduler health: airflow jobs check --job-type SchedulerJob --hostname $(hostname) If it returns Found one alive job. , the scheduler is alive. If not, check logs at $AIRFLOW_HOME/logs/scheduler/latest/ . For the metadata database connectivity: airflow db check Expected output: Connection successful. If there is an error, verify database credentials and network. List DAGs and their states: airflow dags list Sample output: dag_id | filepath | owner | paused ====================+===================+=========+======= example_dag | example_dag.py | airflow | False data_pipeline | data_pipeline.py | data_team | False Pay attention to the paused column. A paused DAG will not run on schedule; unpause with airflow dags unpause data_pipeline . Inspect task instances for a specific DAG run: airflow tasks list data_pipeline Then check the state of a task instance: airflow tasks state data_pipeline transform_data 2025-03-25T00:00:00+00:00 Expected output: success If it is failed , examine logs: airflow tasks logs data_pipeline transform_data 2025-03-25T00:00:00+00:00 For Airflow 3, use the API endpoints for similar diagnostics. For example, GET /api/v2/dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances returns task state details in JSON. ## Failure Modes and Recovery Understand common failure modes and how to recover using commands. ### Scheduler Down If the scheduler is not running, DAGs will not be triggered. Check with airflow jobs check . If no alive job, start the scheduler: airflow scheduler Run it in a supervised process (e.g., systemd, Docker) to auto-restart. ### Metadata Database Unavailable If airflow db check fails, verify the database service is running and credentials are correct. If the database is down, restore from backup or restart the database service. Do not run airflow db migrate unless you are intentionally upgrading; it may change the schema. ### DAG Parse Errors If a DAG file has a syntax error, it will not appear in airflow dags list or will show an import error. Use: export AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT=60 airflow dags list Check the scheduler logs for tracebacks. Fix the syntax, then run: airflow dags reserialize This forces serialization and may reveal errors. ### Task Failure A task may fail due to code bugs, resource limits, or external dependencies. After identifying the failed task instance, you can clear it to re-run: airflow tasks clear data_pipeline --task-regex transform_data --start-date 2025-03-25T00:00:00+00:00 --end-date 2025-03-25T01:00:00+00:00 This clears the task instances in that range and allows the scheduler to re-run them. For a single DAG run, use --dag-run-id . Always check logs before clearing to avoid repeating the same failure. ### Zombie Tasks If a task is stuck in running state but the worker died, mark it as failed: airflow tasks failed data_pipeline transform_data 2025-03-25T00:00:00+00:00 Then clear it if appropriate. ## Operations Checklist Use this checklist for daily or weekly Airflow operations. Each item includes an owner and frequency.
ItemCommand / ActionOwnerFrequency
Check version and environmentairflow version, airflow config get-value core executorPlatform EngineerWeekly
Verify scheduler healthairflow jobs check --job-type SchedulerJobOn-call EngineerDaily
Verify database connectivityairflow db checkOn-call EngineerDaily
List DAGs and paused statusairflow dags listData EngineerDaily
Check for recent task failuresairflow dags list-runs --state failed --start-date $(date -d '1 day ago' +%Y-%m-%dT%H:%M:%S)Data EngineerDaily
Review DAG parse errorsScheduler logs or airflow dags list for missing DAGsData EngineerWeekly
Backup metadata databaseDatabase-specific backup (e.g., pg_dump)DBA / Platform EngineerDaily
Test upgrade/migration on stagingRun migration scripts and sample DAGsPlatform EngineerBefore any upgrade
Audit configuration changesReview airflow.cfg or environment variables against source controlPlatform EngineerMonthly
## Common Pitfalls and How to Avoid Them ### 1. Treating HTTP 200 as Full Health Why it happens: Operators often assume that if the Airflow web server responds, the entire system is healthy. However, the web server may be running while the scheduler is down. How to avoid: Use the health endpoint ( /api/v2/monitor/health in Airflow 3) and check each component's status. Also run airflow jobs check . ### 2. Running airflow db migrate Unnecessarily Why it happens: Some operators believe that running db migrate is a quick fix for database issues. It is intended for schema upgrades and can cause problems if run at the wrong time. How to avoid: Only run db migrate as part of an upgrade procedure, after backing up the database. For connectivity issues, use db check . ### 3. Ignoring Version Differences Why it happens: Commands and behavior change between Airflow versions, especially the jump from 2 to 3. Operators may use old commands that no longer exist or behave differently. How to avoid: Always check airflow version and consult the documentation for your specific version. Test commands in a non-production environment. ### 4. Clearing Task Instances Without Investigating Why it happens: When a task fails, the immediate reaction is to clear it and hope it succeeds. Without understanding the root cause, the task may fail again and waste resources. How to avoid: Examine logs with airflow tasks logs before clearing. Check for code errors, data issues, or resource exhaustion. ### 5. Exposing Secrets in Commands Why it happens: Operators may inadvertently print connection strings or passwords in command output or logs. For example, airflow config list may show some secrets if not redacted. How to avoid: Use the Airflow secrets backend (e.g., environment variables, Vault) and avoid passing credentials as command arguments. Always review command output before sharing. ### 6. Not Backing Up Before Changes Why it happens: In a rush to fix an issue, operators may change configuration or run migrations without a backup. How to avoid: Establish a routine backup of the metadata database and DAG files. For critical changes, snapshot the database manually. ## Conclusion Mastering basic Airflow commands is essential for safe and efficient daily operations. By following a structured approach—inventory, observation, targeted intervention, and verification—you reduce risk and improve reliability. Use the commands and checklists in this guide as a starting point, and always adapt to your specific version and deployment. As a next step, choose one low-risk verification command from this guide, run it against your Airflow environment, and document the current state and expected output. Then review your failure recovery procedures to ensure you are prepared for common incidents.