Integrating Grafana k6 with Prometheus and Grafana: Real-Time API Load Monitoring and SLA/SLO Tracking

Monitoring tutorial - IT technology blog
Monitoring tutorial - IT technology blog

1. The Midnight Crash and the Costly Lessons of “Blind” Load Testing

Six months ago, our team was gearing up for a major Black Friday sale. Everyone was riding high on confidence. QA and backend engineers had simulated 2,000 concurrent users using k6 in our staging environment: a sea of green in the terminal, a 99.8% HTTP 200 success rate, and an average latency of just 85ms. Feeling relieved, everyone closed their laptops and called it a night.

Reality hit us with a harsh wake-up call. At 12:07 AM sharp, real traffic flooded the checkout system. The checkout API choked instantly, with latency skyrocketing to 4.5 seconds. The PostgreSQL connection pool hit its 100-connection ceiling within three minutes, triggering a cascading failure across all downstream background workers. Our fan page was inundated with angry customer complaints.

That night taught me a painful lesson: running a load test and relying on a few lines of terminal summary is like driving blindfolded. You have no idea when the system actually starts buckling. Is your breaking point at 800 or 1,500 RPS? Did you breach your SLA/SLO the second traffic peaked? Without real-time charts, there is simply no way to know.

2. Why k6’s Static Reports Won’t Save Your System

Running k6’s default terminal output is convenient: fire off a command, wait 10 minutes, and get a summary table. However, this approach exposes three dangerous blind spots:

  • Averages hide disasters: An average latency of 80ms sounds reassuring. But in production, tail latency (p95, p99) is what brings systems down. It only takes 1% of requests stalling on a 10-second database lock to starve your worker threads and crash the entire API fleet.
  • Lost correlation between load and infrastructure: Twelve minutes into the test, response times jump 5x. Why? Did CPU max out, was disk I/O saturated, or was the network bottlenecked? k6’s terminal summary sits in isolation, completely detached from server infrastructure metrics.
  • Purely reactive SLO tracking: Suppose your SLO commitment is “99% of requests respond in under 200ms at 2,000 RPS.” If you only find out you breached it after the test finishes, you will waste hours rerunning tests from scratch every time you tweak a line of code.

3. Three Common Approaches and the Cost Equation

Before settling on our final architecture, our team evaluated the standard options:

Option 1: Exporting JSON/CSV Files to Parse and Plot

k6 lets you export raw metrics via the --out json=result.json flag. It’s completely free. On the downside, a single 15-minute test run at 1,000 RPS generates a file over 6GB in size. Opening it with Python scripts or dumping it into Excel is sluggish and laggy—and most importantly, you still can’t monitor progress in real time while the test runs.

Option 2: Purchasing Grafana Cloud k6 SaaS

Grafana Labs’ official managed service is sleek, offering polished dashboards and granular endpoint analytics. However, the subscription price—around $500/month for a team plan—is a steep hurdle for small to mid-sized projects. On top of that, many enterprise clients strictly forbid sending internal performance metrics to external third-party servers due to compliance and security policies.

Option 3: Streaming Metrics Directly into an Existing Prometheus + Grafana Stack

Our internal monitoring stack already had Prometheus and Grafana monitoring 15 server nodes. Leveraging this existing infrastructure proved to be the most cost-effective and powerful move. With zero additional licensing costs, all metrics—from application throughput to server CPU, RAM, and IOPS—converge into a single pane of glass.

4. Hands-on Setup: Streaming k6 Metrics to Prometheus via Remote Write

Here is the standardized pipeline our team has run reliably for the past six months. k6 provides native support for experimental-prometheus-rw, streaming data directly to Prometheus without requiring intermediate StatsD or Pushgateway setups.

Step 1: Enable the Remote Write Receiver in Prometheus

Prometheus operates on a pull (scrape) model by default. To allow Prometheus to accept pushed metrics from k6, you must enable the --web.enable-remote-write-receiver feature flag in your configuration:

# docker-compose.yml configuration for Prometheus
services:
  prometheus:
    image: prom/prometheus:v2.45.0
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.enable-remote-write-receiver'
      - '--storage.tsdb.retention.time=15d'
    ports:
      - "9090:9090"

Restart the container. The write endpoint will now be available at http://<prometheus-ip>:9090/api/v1/write.

Step 2: Write a k6 Script Tied Directly to SLA/SLO Targets

We configure a ramp-up workload stage and declare thresholds matching our system’s actual SLOs:

import http from 'k6/http';
import { check, sleep } from 'k6';

// Load scenario and SLA/SLO targets
export const options = {
  stages: [
    { duration: '1m', target: 50 },  // First 1m: ramp up to 50 VUs
    { duration: '3m', target: 200 }, // Next 3m: sustain at 200 VUs
    { duration: '1m', target: 0 },   // Final 1m: ramp down to 0
  ],
  thresholds: {
    // SLO 1: Total test error rate must be under 1%
    http_req_failed: ['rate<0.01'],
    // SLO 2: 95% of requests must respond under 200ms
    http_req_duration: ['p(95)<200'],
    // SLO 3: 99% of successful requests must be under 500ms
    'http_req_duration{status:200}': ['p(99)<500'],
  },
};

export default function () {
  const res = http.get('http://api-internal.example.com/v1/products');
  
  check(res, {
    'status 200 OK': (r) => r.status === 200,
    'has data returned': (r) => r.body.includes('items'),
  });

  sleep(1);
}

Step 3: Execute the Load Test with Remote Write Output

Set the environment variable pointing to your Prometheus URL and run the test:

K6_PROMETHEUS_REMOTE_URL=http://192.168.1.100:9090/api/v1/write \
k6 run -o experimental-prometheus-rw load_test.js

k6 batches metrics and flushes them directly to Prometheus every few seconds. You can now open Grafana to watch your performance curves update in real time.

Step 4: Configure an SLO Monitoring Dashboard in Grafana

You can import community dashboards (such as ID 18030 or 14353), or build your own panels with these three core PromQL queries:

  • Actual Throughput (RPS):
    sum(rate(k6_http_reqs_total[30s]))
  • p95 and p99 Latency vs. SLO Targets:
    histogram_quantile(0.95, sum(rate(k6_http_req_duration_seconds_bucket[1m])) by (le)) * 1000
  • Error Rate Percentage (%):
    sum(rate(k6_http_req_failed_total{expected="false"}[30s])) / sum(rate(k6_http_reqs_total[30s])) * 100

5. Hard-Won Lessons from Six Months in Production

Across dozens of stress tests, here are two crucial takeaways to keep in mind:

  1. Watch out for Prometheus cardinality explosion: Never attach high-cardinality labels (such as user_id, order_id, or dynamic URLs like /items/12345) to custom k6 metrics. A test with 100,000 virtual users will generate millions of distinct time series, exhausting Prometheus memory and crashing it instantly. Always normalize dynamic endpoints into patterns like /items/{id} using http.url.
  2. Use a split-screen view during tests: Position your k6 p95 Latency panel directly alongside your Database CPU, RAM, and Connection Pool metrics. You will immediately spot correlations: whenever Postgres connection pool usage hits 80%, API p95 latency spikes vertically. Pinpointing bottlenecks takes 30 seconds rather than an entire afternoon digging through logs.
Share: