Monitoring Apache Airflow with Prometheus and Grafana: Tracking DAG Runs, Task Failures, and Scheduler Health

Context: Why You Shouldn’t Rely Solely on the Airflow Webserver

If you’ve ever operated data pipelines, you probably know that anxious feeling every morning. Your Airflow cluster carries hundreds of DAGs running silently throughout the night—from pulling CRM data and syncing orders to the Data Warehouse, to aggregating revenue numbers at 6:00 AM right in time for the daily briefing.

Everything runs smoothly until 2:00 AM. An extraction task fails due to a database connection timeout on the partner’s side. Or worse: the Airflow Scheduler runs completely out of memory and crashes. The queue backs up, downstream jobs stall, and no alarms sound. At 8:30 AM sharp, when leadership opens a blank dashboard, your phone starts blowing up.

Many new engineers rely too heavily on the default Airflow web interface. The webserver is great for inspecting individual tasks, but it leaves you practically blind when managing the entire cluster. You can’t observe resource trends over the past 30 days.

Nor can you measure average scheduler latency when your DAG count doubles. Most dangerously: when workers or schedulers crash, the webserver often goes down with them. To end the nightmare of SSHing into nodes in the middle of the night, the standard solution is exporting metrics from Airflow to Prometheus and visualizing them via Grafana.

Monitoring Architecture: Why Do We Need StatsD Exporter?

Prometheus operates on a pull model over HTTP. Airflow, on the other hand, constantly generates events and pushes metrics out via the StatsD protocol using UDP packets. UDP packets run asynchronously, are extremely lightweight, and won’t interrupt task ETL execution. To allow Prometheus to ingest this data, we need an intermediary: Prometheus StatsD Exporter.

The workflow operates in 3 steps:

  • Airflow emits plaintext metrics over UDP port 8125 to StatsD Exporter.
  • StatsD Exporter receives, aggregates, and transforms metric strings into standard Prometheus labeled metrics.
  • Prometheus periodically scrapes these standardized metrics over HTTP port 9102.

Here is the complete docker-compose.monitoring.yml file to spin up the entire monitoring stack:

version: '3.8'

services:
  statsd-exporter:
    image: prom/statsd-exporter:v0.26.0
    container_name: airflow_statsd_exporter
    restart: always
    command:
      - "--statsd.mapping-config=/etc/statsd-exporter/statsd.conf"
    ports:
      - "8125:8125/udp"
      - "9102:9102"
    volumes:
      - ./statsd.conf:/etc/statsd-exporter/statsd.conf

  prometheus:
    image: prom/prometheus:v2.50.0
    container_name: prometheus
    restart: always
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus

  grafana:
    image: grafana/grafana:10.3.0
    container_name: grafana
    restart: always
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana

volumes:
  prometheus_data:
  grafana_data:

Step-by-Step Configuration: Connecting Airflow and Prometheus

Step 1: Enable Metrics Emission in Airflow

In your airflow.cfg configuration file, locate the [metrics] section and update it:

[metrics]
statsd_on = True
statsd_host = statsd-exporter
statsd_port = 8125
statsd_prefix = airflow

If you deploy Airflow on Docker Compose or Kubernetes (via Helm chart), pass these settings directly using environment variables:

AIRFLOW__METRICS__STATSD_ON=True
AIRFLOW__METRICS__STATSD_HOST=statsd-exporter
AIRFLOW__METRICS__STATSD_PORT=8125
AIRFLOW__METRICS__STATSD_PREFIX=airflow

Step 2: Standardize Labels with statsd.conf

By default, Airflow outputs metrics as dot-delimited strings (e.g., airflow.dagrun.duration.success.sync_orders_hourly). Prometheus needs these dimensions separated into distinct labels for flexible querying. The statsd.conf file handles this mapping:

mappings:
  - match: "airflow.dagrun.duration.success.*"
    name: "airflow_dagrun_duration_success_seconds"
    labels:
      dag_id: "$1"

  - match: "airflow.dagrun.duration.failed.*"
    name: "airflow_dagrun_duration_failed_seconds"
    labels:
      dag_id: "$1"

  - match: "airflow.ti_failures.*.*"
    name: "airflow_task_instance_failures_total"
    labels:
      dag_id: "$1"
      task_id: "$2"

  - match: "airflow.dag_processing.last_duration.*"
    name: "airflow_dag_processing_duration_seconds"
    labels:
      dag_file: "$1"

Step 3: Define Scrape Targets in prometheus.yml

Add the following configuration to prometheus.yml so Prometheus scrapes statsd-exporter every 15 seconds:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'airflow-statsd'
    static_configs:
      - targets: ['statsd-exporter:9102']

Verifying Connectivity and Building Alerting Dashboards

Validating Collected Metrics

Start the container stack with docker compose up -d. After triggering a few test tasks, verify the exporter endpoint directly:

curl -s http://localhost:9102/metrics | grep airflow

If the terminal prints entries matching the format below, your metrics pipeline is working properly:

airflow_task_instance_failures_total{dag_id="sync_orders_hourly",task_id="extract_mysql"} 1

4 Core Metrics to Monitor in Grafana

Log in to Grafana on port 3000 with admin/admin credentials. After adding the Prometheus Data Source (http://prometheus:9090), prioritize creating panels for these 4 key metric groups:

1. Scheduler Heartbeat

The scheduler orchestrates the entire pipeline. When this process hangs, no new tasks can be scheduled or triggered.

# Time elapsed since the last heartbeat (in seconds)
time() - airflow_scheduler_heartbeat

Alert Threshold: If the value exceeds 30 seconds, the scheduler is showing signs of overload. If it exceeds 60 seconds, the process is practically dead and should trigger an immediate alert to Slack or PagerDuty.

2. Task Failure Rate

Catch tasks that frequently fail across execution cycles before downstream impact occurs:

# Task failure count over the past 15-minute window
sum by (dag_id, task_id) (rate(airflow_task_instance_failures_total[15m])) > 0

This metric helps you immediately detect cascading failures caused by exhausted third-party API quotas or database deadlock issues.

3. DAG Run Duration

Execution duration graphs directly reflect pipeline health and changes in input data volume:

airflow_dagrun_duration_success_seconds

If an ETL pipeline that normally takes 5 minutes suddenly spikes to 40 minutes, investigate immediately for data volume spikes or missing indexes in SQL queries.

4. Worker Queue and Capacity (Queued Tasks & Open Slots)

Monitor pending tasks alongside available slots across your workers:

# Number of tasks currently stuck in queue
airflow_executor_queued_tasks

# Remaining open slots ready to accept work
airflow_executor_open_slots

Signs of a Bottleneck: When airflow_executor_open_slots stays at 0 for more than 10 minutes while airflow_executor_queued_tasks surges, your worker pool is out of resources. You need to scale out additional worker pods or tune the parallelism configuration.

Setting up comprehensive monitoring allows data teams to transition from reactive troubleshooting to proactively preempting bottlenecks, ensuring critical business data is always ready on schedule.

Share: