The Real Problem
I was running Traefik as a reverse proxy for about 8 containers on a VPS. Everything ran fine for the first two weeks. Then one morning, users reported the site was unusually slow — containers were still running, Traefik showed no obvious errors. I SSH’d into the server, ran docker stats, and read through logs container by container. It took nearly 30 minutes to track down the bottleneck: a service that was taking too long to process requests.
The problem wasn’t a lack of tools. The problem was that Traefik was operating as a black box: you knew it was running, but had no idea how many requests were flowing through, how many were returning 5xx errors, or what the average latency was in milliseconds. Without those numbers, debugging was just guesswork.
What Visibility Does Traefik Lack?
By default, Traefik doesn’t expose any metrics. When containers are healthy, all you see are text logs — no data to graph over time. Here’s what’s missing:
- Request Rate: How many requests per second are flowing through each entrypoint or router?
- Error Rate: What’s the 4xx/5xx response ratio? Which service is returning the most errors?
- Percentile Latency: What are the P50, P95, P99 latency values for each backend service?
- Active Connections: How many TCP connections are open at any given moment?
Traefik supports exporting metrics to Prometheus, Datadog, or InfluxDB — but all of these are disabled by default. The two approaches below range from quick-and-dirty to fully featured, depending on your actual needs.
Solutions
Option 1: Enable Access Logs and Analyze Manually
The fastest option for immediate debugging. Add this to traefik.yml:
accessLog:
filePath: "/var/log/traefik/access.log"
format: json
fields:
headers:
defaultMode: dropCount 5xx errors:
grep '"DownstreamStatus":5' /var/log/traefik/access.log | wc -lUseful for quick one-off debugging. But no dashboard, no alerts, no trend visibility over time — not suitable for long-term production monitoring.
Option 2: Prometheus Metrics + Grafana
Traefik ships with a built-in /metrics endpoint in Prometheus format — enable it, let Prometheus scrape it, build a Grafana dashboard, and you’re done. Takes about 30 minutes to set up and lasts indefinitely. This is what I’m running in production.
Full Setup with Docker Compose
Step 1: Configure Traefik to Expose Prometheus Metrics
Add the metrics section to traefik.yml. Note: addRoutersLabels and addServicesLabels are the two most commonly forgotten options — without them, metrics only aggregate by entrypoint and can’t be broken down per service:
# traefik.yml
api:
dashboard: true
insecure: true
entryPoints:
web:
address: ":80"
websecure:
address: ":443"
metrics:
address: ":8082"
metrics:
prometheus:
entryPoint: metrics
addEntryPointsLabels: true
addRoutersLabels: true
addServicesLabels: true
buckets:
- 0.1
- 0.3
- 1.2
- 5.0
providers:
docker:
exposedByDefault: falseStep 2: Docker Compose for the Full Stack
The docker-compose.yml for Traefik + Prometheus + Grafana:
version: '3.8'
networks:
proxy:
external: true
monitoring:
internal: true
volumes:
prometheus_data:
grafana_data:
services:
traefik:
image: traefik:v3.0
container_name: traefik
restart: unless-stopped
ports:
- "80:80"
- "443:443"
- "8080:8080" # dashboard
- "8082:8082" # metrics
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./traefik.yml:/traefik.yml:ro
networks:
- proxy
- monitoring
prometheus:
image: prom/prometheus:v2.51.0
container_name: prometheus
restart: unless-stopped
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.retention.time=15d'
networks:
- monitoring
grafana:
image: grafana/grafana:10.4.0
container_name: grafana
restart: unless-stopped
environment:
- GF_SECURITY_ADMIN_PASSWORD=your_strong_password
- GF_USERS_ALLOW_SIGN_UP=false
volumes:
- grafana_data:/var/lib/grafana
networks:
- monitoring
- proxy
labels:
- "traefik.enable=true"
- "traefik.http.routers.grafana.rule=Host(`grafana.yourdomain.com`)"
- "traefik.http.routers.grafana.entrypoints=websecure"Step 3: Configure Prometheus to Scrape Traefik
The prometheus.yml file. Since Traefik and Prometheus share the monitoring network, use the service name instead of an IP address — no need to hardcode anything:
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "traefik_alerts.yml"
scrape_configs:
- job_name: 'traefik'
static_configs:
- targets: ['traefik:8082']
metrics_path: /metricsVerify that Traefik is exposing metrics:
curl http://localhost:8082/metrics | grep traefik_entrypointSample output:
traefik_entrypoint_requests_total{code="200",entrypoint="websecure",method="GET",protocol="http"} 1523
traefik_entrypoint_requests_total{code="404",entrypoint="websecure",method="GET",protocol="http"} 42
traefik_entrypoint_request_duration_seconds_bucket{entrypoint="websecure",le="0.1"} 1401Step 4: PromQL Queries for the Grafana Dashboard
Add the Prometheus datasource to Grafana (http://prometheus:9090), then create a dashboard with these panels:
Request Rate — total requests per second by entrypoint:
sum(rate(traefik_entrypoint_requests_total[2m])) by (entrypoint)Error Rate — percentage of 5xx requests by service:
100 * sum(rate(traefik_service_requests_total{code=~"5.."}[2m])) by (service)
/
sum(rate(traefik_service_requests_total[2m])) by (service)P95 Latency — 95th percentile latency by service:
histogram_quantile(0.95,
sum(rate(traefik_service_request_duration_seconds_bucket[5m])) by (le, service)
)Active connections by entrypoint:
traefik_entrypoint_open_connections{entrypoint="websecure"}Don’t want to build from scratch? Import Grafana dashboard ID 17346 — the official Traefik dashboard, ready to use out of the box, customize as needed.
Step 5: Alert Rules for Automated Notifications
Create traefik_alerts.yml in the same directory as prometheus.yml:
groups:
- name: traefik_alerts
rules:
- alert: TraefikHighErrorRate
expr: |
100 * sum(rate(traefik_service_requests_total{code=~"5.."}[5m])) by (service)
/
sum(rate(traefik_service_requests_total[5m])) by (service)
> 5
for: 2m
labels:
severity: warning
annotations:
summary: "Service {{ $labels.service }} error rate exceeded 5%"
description: "Current error rate: {{ $value | printf \"%.1f\" }}%"
- alert: TraefikHighLatency
expr: |
histogram_quantile(0.95,
sum(rate(traefik_service_request_duration_seconds_bucket[5m])) by (le, service)
) > 2
for: 5m
labels:
severity: warning
annotations:
summary: "P95 latency of {{ $labels.service }} exceeded 2 seconds"Real-World Results
Once the dashboard was up, everything changed noticeably. Request rate per service, error rate when a backend container crashes, latency breakdown by P50/P95/P99 — all of it live on a single screen. No more SSH sessions. I often pull up the dashboard on my phone while out and catch issues before users even have a chance to report them.
One thing to keep in mind: traefik_service_* metrics only appear after at least one real request has passed through that service. Grafana panels showing “No data” right after setup is expected — send a few test requests first:
curl -s https://yourdomain.com/healthz
# Wait 15-30 seconds, then check Grafana againOn resource usage: Prometheus + Grafana consumes around 300MB of RAM — fine for a 2GB VPS. --storage.tsdb.retention.time=15d keeps 15 days of data, enough to spot trends and trace back incidents that happened over the weekend before anyone noticed.
