Advanced PromQL: Don’t Let “Average Numbers” Mislead You

The Pitfall of “Average” Numbers

When I first started in DevOps, I used to be thrilled looking at a Grafana dashboard showing an average latency of 100ms. Everything was green, and I was convinced the system was running perfectly. Suddenly, the team’s Support group chat exploded: “Customers are complaining that the payment process is lagging; it’s just spinning forever!”.

That’s when I realized a fatal mistake. That 100ms average was actually the result of 90 users with lightning-fast response times (10ms) and 10 others waiting for a full 2 seconds. Those 10 people were the angry customers. If you only use basic up or rate commands, you’ll stay trapped in the illusion of beautiful but meaningless numbers.

To find the truth, we need heavy-duty “weapons” in PromQL. Join me as we dissect how to use Histograms and Subqueries to see through the system.

Why Basic Queries Often Miss Errors?

Commands like rate(http_requests_total[5m]) only tell you the current request rate. It is completely blind to outliers.

In reality, there are three issues that make Junior dashboards unreliable:

• Overusing Averages: Arithmetic means blur out dangerous spikes. A server that is stalled for 10 seconds can be hidden by 50 minutes of stable operation.
• Losing Time Traces: It is hard to identify the highest peak of error rates over the last 24 hours if you only look at real-time data.
• Data Noise: When running hundreds of microservices, indiscriminate sum operations make it impossible to pinpoint which specific instance is “unhealthy.”

Histogram Quantile: Measuring Real-World Experience (P95, P99)

This is my top technique for measuring latency. Instead of calculating averages, histogram_quantile helps you know exactly what speed 99% of users are experiencing.

Suppose you have the metric http_request_duration_seconds_bucket. To calculate P99 (99% of requests completed within this range or faster), use:

histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

The mechanism is very simple:

• rate(...[5m]): Calculates the growth rate of buckets over 5 minutes.
• sum by (le): Aggregates data by the le (less than or equal) label. This is the vital label for Histograms.
• 0.99: The percentile threshold you want to examine.

If the result is 0.5, it means 99% of users receive responses in under 500ms. If this number spikes to 2s while the average remains 100ms, you know for sure the system has a serious problem for a specific group of users.

Subquery: Queries Within Queries

Have you ever wondered: “In the last hour, was there any point where the error rate exceeded 5%?”. If you use a regular rate, you only see the value at this exact moment. Subqueries help you scan the past like a dynamic data array.

Basic syntax: <query> [<range>:<resolution>].

For example, to find the maximum error rate (5-minute window) that occurred within the last hour:

max_over_time(
  rate(http_requests_total{status=~"5.."}[5m])[1h:1m]
)

In this query, [1h:1m] means scanning data from 1 hour ago, taking a sample every 1 minute. This technique is extremely useful for setting up alerts. It helps avoid “flapping” alerts when a metric just ticks up slightly and then immediately drops.

Smart Aggregation with “without”

Instead of using sum by (instance, pod, region) and having to list everything out, try using without. It allows you to remove unnecessary labels and keep everything else.

sum without (instance, pod) (rate(http_requests_total[5m]))

This command sums up requests but keeps labels like method or endpoint. This makes your query much more flexible. When the system adds new labels like az or version, the query updates automatically without you needing to change the code.

Tips for Professional Dashboard Optimization

To make Dashboards not just beautiful but also fast, I usually apply three golden rules:

1. Use Recording Rules: Commands like histogram_quantile are very CPU-intensive. Let Prometheus pre-calculate and store them in a new metric instead of forcing it to recalculate every time you refresh your browser.
2. P50 – P95 – P99 Combo: Always display these three numbers together. The gap between them reveals the system’s stability. If they are far apart, the system is experiencing local bottlenecks.
3. Control Cardinality: Never assign labels with constantly changing values like user_id. This will cause Prometheus data to bloat and crash the system in no time.

Mastering PromQL is a big step from someone who just “looks at charts” to an engineer capable of “reading” the system. Since applying P99 and Subqueries, I no longer have to stay up at night SSHing into servers to guess what’s wrong. Everything is now clearly displayed through numbers that speak for themselves.