Context: Why You Shouldn’t Rely Solely on the Airflow Webserver
If you’ve ever operated data pipelines, you probably know that anxious feeling every morning. Your Airflow cluster carries hundreds of DAGs running silently throughout the night—from pulling CRM data and syncing orders to the Data Warehouse, to aggregating revenue numbers at 6:00 AM right in time for the daily briefing.
Everything runs smoothly until 2:00 AM. An extraction task fails due to a database connection timeout on the partner’s side. Or worse: the Airflow Scheduler runs completely out of memory and crashes. The queue backs up, downstream jobs stall, and no alarms sound. At 8:30 AM sharp, when leadership opens a blank dashboard, your phone starts blowing up.
Many new engineers rely too heavily on the default Airflow web interface. The webserver is great for inspecting individual tasks, but it leaves you practically blind when managing the entire cluster. You can’t observe resource trends over the past 30 days.
Nor can you measure average scheduler latency when your DAG count doubles. Most dangerously: when workers or schedulers crash, the webserver often goes down with them. To end the nightmare of SSHing into nodes in the middle of the night, the standard solution is exporting metrics from Airflow to Prometheus and visualizing them via Grafana.
Monitoring Architecture: Why Do We Need StatsD Exporter?
Prometheus operates on a pull model over HTTP. Airflow, on the other hand, constantly generates events and pushes metrics out via the StatsD protocol using UDP packets. UDP packets run asynchronously, are extremely lightweight, and won’t interrupt task ETL execution. To allow Prometheus to ingest this data, we need an intermediary: Prometheus StatsD Exporter.
The workflow operates in 3 steps:
- Airflow emits plaintext metrics over UDP port 8125 to StatsD Exporter.
- StatsD Exporter receives, aggregates, and transforms metric strings into standard Prometheus labeled metrics.
- Prometheus periodically scrapes these standardized metrics over HTTP port 9102.
Here is the complete docker-compose.monitoring.yml file to spin up the entire monitoring stack:
version: '3.8'
services:
statsd-exporter:
image: prom/statsd-exporter:v0.26.0
container_name: airflow_statsd_exporter
restart: always
command:
- "--statsd.mapping-config=/etc/statsd-exporter/statsd.conf"
ports:
- "8125:8125/udp"
- "9102:9102"
volumes:
- ./statsd.conf:/etc/statsd-exporter/statsd.conf
prometheus:
image: prom/prometheus:v2.50.0
container_name: prometheus
restart: always
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus_data:/prometheus
grafana:
image: grafana/grafana:10.3.0
container_name: grafana
restart: always
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- grafana_data:/var/lib/grafana
volumes:
prometheus_data:
grafana_data:
Step-by-Step Configuration: Connecting Airflow and Prometheus
Step 1: Enable Metrics Emission in Airflow
In your airflow.cfg configuration file, locate the [metrics] section and update it:
[metrics]
statsd_on = True
statsd_host = statsd-exporter
statsd_port = 8125
statsd_prefix = airflow
If you deploy Airflow on Docker Compose or Kubernetes (via Helm chart), pass these settings directly using environment variables:
AIRFLOW__METRICS__STATSD_ON=True
AIRFLOW__METRICS__STATSD_HOST=statsd-exporter
AIRFLOW__METRICS__STATSD_PORT=8125
AIRFLOW__METRICS__STATSD_PREFIX=airflow
Step 2: Standardize Labels with statsd.conf
By default, Airflow outputs metrics as dot-delimited strings (e.g., airflow.dagrun.duration.success.sync_orders_hourly). Prometheus needs these dimensions separated into distinct labels for flexible querying. The statsd.conf file handles this mapping:
mappings:
- match: "airflow.dagrun.duration.success.*"
name: "airflow_dagrun_duration_success_seconds"
labels:
dag_id: "$1"
- match: "airflow.dagrun.duration.failed.*"
name: "airflow_dagrun_duration_failed_seconds"
labels:
dag_id: "$1"
- match: "airflow.ti_failures.*.*"
name: "airflow_task_instance_failures_total"
labels:
dag_id: "$1"
task_id: "$2"
- match: "airflow.dag_processing.last_duration.*"
name: "airflow_dag_processing_duration_seconds"
labels:
dag_file: "$1"
Step 3: Define Scrape Targets in prometheus.yml
Add the following configuration to prometheus.yml so Prometheus scrapes statsd-exporter every 15 seconds:
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'airflow-statsd'
static_configs:
- targets: ['statsd-exporter:9102']
Verifying Connectivity and Building Alerting Dashboards
Validating Collected Metrics
Start the container stack with docker compose up -d. After triggering a few test tasks, verify the exporter endpoint directly:
curl -s http://localhost:9102/metrics | grep airflow
If the terminal prints entries matching the format below, your metrics pipeline is working properly:
airflow_task_instance_failures_total{dag_id="sync_orders_hourly",task_id="extract_mysql"} 1
4 Core Metrics to Monitor in Grafana
Log in to Grafana on port 3000 with admin/admin credentials. After adding the Prometheus Data Source (http://prometheus:9090), prioritize creating panels for these 4 key metric groups:
1. Scheduler Heartbeat
The scheduler orchestrates the entire pipeline. When this process hangs, no new tasks can be scheduled or triggered.
# Time elapsed since the last heartbeat (in seconds)
time() - airflow_scheduler_heartbeat
Alert Threshold: If the value exceeds 30 seconds, the scheduler is showing signs of overload. If it exceeds 60 seconds, the process is practically dead and should trigger an immediate alert to Slack or PagerDuty.
2. Task Failure Rate
Catch tasks that frequently fail across execution cycles before downstream impact occurs:
# Task failure count over the past 15-minute window
sum by (dag_id, task_id) (rate(airflow_task_instance_failures_total[15m])) > 0
This metric helps you immediately detect cascading failures caused by exhausted third-party API quotas or database deadlock issues.
3. DAG Run Duration
Execution duration graphs directly reflect pipeline health and changes in input data volume:
airflow_dagrun_duration_success_seconds
If an ETL pipeline that normally takes 5 minutes suddenly spikes to 40 minutes, investigate immediately for data volume spikes or missing indexes in SQL queries.
4. Worker Queue and Capacity (Queued Tasks & Open Slots)
Monitor pending tasks alongside available slots across your workers:
# Number of tasks currently stuck in queue
airflow_executor_queued_tasks
# Remaining open slots ready to accept work
airflow_executor_open_slots
Signs of a Bottleneck: When airflow_executor_open_slots stays at 0 for more than 10 minutes while airflow_executor_queued_tasks surges, your worker pool is out of resources. You need to scale out additional worker pods or tune the parallelism configuration.
Setting up comprehensive monitoring allows data teams to transition from reactive troubleshooting to proactively preempting bottlenecks, ensuring critical business data is always ready on schedule.