Production Incident: When Varnish Cache Becomes a “Black Box”
During last year’s mid-year sale, our website traffic surged from 1,200 to nearly 6,500 RPS. Our four-node Nginx/PHP-FPM backend cluster immediately spiked to 100% CPU utilization. The load average shot up to 48, while the frontend Varnish node was barely consuming 2.5 GB out of its 8 GB of RAM.
I SSH’d into the server and ran varnishstat. Metrics were fluctuating wildly second by second, making it nearly impossible to spot trends. Varnish appeared to be handling connections normally. However, a VCL snippet deployed that morning had failed to strip a new tracking cookie introduced by the marketing team. As a result, over 80% of all requests were bypassed straight to the backend (pass mode). The Cache Hit Rate plunged from 94% down to 15% without anyone noticing.
With no visual dashboards and no automated alerts, our engineering team was left completely in the dark. To solve this once and for all, we implemented a monitoring stack using prometheus-varnish-exporter, Prometheus, and Grafana. Here is our monitoring architecture, step-by-step setup guide, and practical battle-tested tips for alert tuning.
Architecture: Collecting Metrics from Varnish Shared Memory
Varnish does not write stats to text log files to prevent I/O bottlenecks. Instead, all runtime performance counters reside directly in VSM (Varnish Shared Memory), typically mounted under /var/lib/varnish/. The varnishstat CLI utility reads from this exact shared memory segment.
Because Prometheus cannot natively query VSM segments, we use prometheus-varnish-exporter as a bridge. The exporter uses the libvarnishapi library to extract performance counters and exposes an HTTP endpoint (defaulting to port 9131) that Prometheus can periodically scrape.
Three Critical Metric Categories to Track
- Cache Hit Rate: Calculated using the counters
varnish_main_cache_hitandvarnish_main_cache_miss. This metric reflects the effectiveness of your VCL caching rules. - Backend Health: Measured via
varnish_backend_conn(active connections to Nginx/Apache) andvarnish_backend_fail(backend timeouts or connection failures). A sudden spike in failures indicates that your backend is overwhelmed. - Client Traffic and Connections: Tracked through
varnish_main_client_req(total incoming requests) andvarnish_main_sess_drop(sessions dropped due to TCP listen queue exhaustion).
Step-by-Step Installation and Configuration
Step 1: Deploy prometheus-varnish-exporter
The exporter is written in Go but links against Varnish’s C libraries. On Ubuntu 22.04 LTS (running Varnish 6.0 LTS or 7.x), install the required library package first:
# Install Varnish API development libraries
sudo apt-get update
sudo apt-get install -y libvarnishapi-dev
# Download the latest binary from GitHub jonnenauha/prometheus_varnish_exporter
cd /tmp
wget https://github.com/jonnenauha/prometheus_varnish_exporter/releases/download/1.6.1/prometheus_varnish_exporter-1.6.1.linux-amd64.tar.gz
tar -xvf prometheus_varnish_exporter-1.6.1.linux-amd64.tar.gz
sudo mv prometheus_varnish_exporter-1.6.1.linux-amd64/prometheus_varnish_exporter /usr/local/bin/
sudo chmod +x /usr/local/bin/prometheus_varnish_exporter
Next, create a systemd unit file. The exporter requires read access to Varnish’s VSM files, so the service should run under the varnish user or group:
sudo tee /etc/systemd/system/varnish_exporter.service << 'EOF'
[Unit]
Description=Prometheus Varnish Exporter
After=varnish.service
[Service]
User=varnish
Group=varnish
ExecStart=/usr/local/bin/prometheus_varnish_exporter -web.listen-address=:9131
Restart=always
RestartSec=3s
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now varnish_exporter.service
Verify that the endpoint is serving metrics using curl:
curl -s http://localhost:9131/metrics | grep -E "varnish_main_cache_hit|varnish_main_client_req"
Step 2: Add Scrape Target to Prometheus
Add a new scrape job configuration to /etc/prometheus/prometheus.yml on your Prometheus server:
scrape_configs:
- job_name: 'varnish_nodes'
scrape_interval: 15s
static_configs:
- targets: ['192.168.1.50:9131']
labels:
environment: 'production'
service: 'varnish-frontend'
Reload the Prometheus configuration: curl -X POST http://localhost:9090/-/reload.
Step 3: Essential PromQL Queries
Varnish counters increase monotonically over the lifetime of the process. Because of this, you should use the rate() function to calculate per-second rates of change instead of using raw counter values.
1. 5-Minute Average Cache Hit Rate (%)
(sum(rate(varnish_main_cache_hit[5m]))
/
(sum(rate(varnish_main_cache_hit[5m])) + sum(rate(varnish_main_cache_miss[5m])))) * 100
Note: This calculation excludes requests deliberately marked as pass or pipe in your VCL, focusing exclusively on cacheable traffic efficiency.
2. Total Request Traffic (RPS) to Varnish
sum(rate(varnish_main_client_req[1m]))
3. Backend Connection Failure Rate
sum(rate(varnish_backend_fail[2m]))
Step 4: Preventing Alert Fatigue
When we first rolled this out, we created a naive rule: whenever the Hit Rate dropped below 85% for 1 minute, fire an alert to Telegram. The result was chaos. Whenever SEO bots crawled the site at night or a developer purged the product catalog cache, Telegram erupted with notifications. On-call engineers became exhausted and started ignoring alerts altogether.
To ensure alerts remain actionable, these three rules made an enormous difference:
- Widen the evaluation window: Evaluate trends over
[10m]rather than[1m]. This smooths out expected temporary dips during cache purges or warmups. - Add a minimum traffic threshold: Only trigger the alert when the Hit Rate is low and overall traffic exceeds a meaningful baseline. For example:
Hit Rate < 70% AND rate(varnish_main_client_req[5m]) > 100. If you get 2 misses out of 4 requests at 3 AM, a 50% hit rate is completely normal and should not wake anyone up. - Separate severity tiers: A 15% drop in Hit Rate should be routed as a Warning via email or a tracking ticket. However, if
varnish_backend_fail > 0persists for over 2 minutes alongsidevarnish_main_sess_drop > 0, that indicates a Critical outage requiring an immediate PagerDuty page.
Bước 5: Building Dashboards in Grafana
In the Grafana interface, set up three essential panel groups:
- Gauge Panel: Displays the Current Hit Ratio. Use distinct threshold color ranges: Red (< 70%), Yellow (70% – 85%), and Green (> 85%).
- Time Series Panel: Plot
varnish_main_cache_hitandvarnish_main_cache_missside by side to observe correlations across peak traffic hours. - Stat Panel: Monitor Session Drops and Backend Unhealthy status. Under normal operating conditions, these numbers should strictly remain at 0.
If you prefer not to build dashboards from scratch, you can import dashboard ID 12431 from Grafana Community Dashboards. Once imported, map it to your Prometheus data source to immediately view your metrics.
Conclusion
Varnish excels at handling thousands of requests per second, but only when you have deep visibility into its traffic patterns. Just like monitoring Memcached to prevent blind spots from overloading your database, combining Prometheus with Grafana empowers your team to spot cache bypasses, backend bottlenecks, and dropped sessions long before users start refreshing and complaining.
Deploy the exporter and monitor your system’s baseline for roughly two weeks. Once you understand your normal operational profile, set up alert thresholds to keep both your infrastructure protected and your on-call team sane.
