A Guide to Using prometheus-client in Python: Collecting and Exporting Custom Metrics for Prometheus and Grafana - ITFROMZERO - Share tobe shared! A detailed guide on using prometheus-client in Python to track Counter, Gauge, and Histogram metrics, avoid high cardinality traps, and build practical Grafana dashboards.

Table of Contents

Context & Why It Matters

When running scripts or backend services in production, nothing is more frustrating than a system that grinds to a halt while logs remain completely silent. Logs only tell you what happened at a specific point in time. They are virtually useless when you need to know current throughput, p99 latency, or how many database connections are bottlenecked.

In one of my previous projects, a cron job synchronizing 50,000 records would randomly freeze in the middle of the night. Sifting through a 2+ GB log file with grep for hours gave no clue whether the script was stuck reading the partner API or writing to the database. After instrumenting the code with prometheus-client, the problem became immediately obvious. Processing throughput dropped from 120 records/sec to 0, while the database connection pool gauge maxed out. No more guesswork.

The Four Core Metric Types to Know

Counter: A cumulative metric that only increases and only resets to zero upon application restart. Perfect for tracking total requests, completed transactions, or failed jobs.
Gauge: A value that can arbitrarily go up or down. Typically used to measure current CPU/memory usage, active worker counts, or available connection pool slots.
Histogram: Measures the distribution of data (response times, payload sizes) by sampling observations into configurable buckets. Enables flexible quantile calculations (p50, p95, p99) directly on the Prometheus server.
Summary: Also samples observations and calculates quantiles directly on the client side. The downside: higher client overhead and impossible to aggregate across multiple instances. In most backend scenarios, Histograms are preferred.

Environment Setup

Install the official library from PyPI:

pip install prometheus-client

To test the end-to-end pipeline with Prometheus and Grafana locally, Docker Compose is the fastest approach. Create a docker-compose.yml file:

version: '3.8'
services:
  prometheus:
    image: prom/prometheus:v2.51.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    extra_hosts:
      - "host.docker.internal:host-gateway"

  grafana:
    image: grafana/grafana:10.4.0
    ports:
      - "3000:3000"
    depends_on:
      - prometheus

Note: The extra_hosts directive above allows the Prometheus container on Linux to connect back to the Python script running on the host machine via the host.docker.internal hostname.

Implementing in Python

1. Defining and Updating Metrics

The start_http_server function spawns a dedicated background daemon thread to serve an HTTP endpoint at /metrics for Prometheus to scrape periodically.

import time
import random
from prometheus_client import start_http_server, Counter, Gauge, Histogram

# Define metrics
REQUEST_COUNT = Counter(
    'app_http_requests_total',
    'Total HTTP requests received by the application',
    ['method', 'endpoint', 'status']
)

ACTIVE_WORKERS = Gauge(
    'app_active_workers',
    'Number of workers currently processing jobs concurrently'
)

REQUEST_LATENCY = Histogram(
    'app_request_latency_seconds',
    'Request processing latency in seconds',
    ['endpoint'],
    buckets=[0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)

def process_request():
    endpoint = random.choice(['/api/v1/orders', '/api/v1/users', '/api/v1/checkout'])
    method = 'POST' if endpoint == '/api/v1/checkout' else 'GET'
    
    ACTIVE_WORKERS.inc()
    start_time = time.time()
    
    try:
        # Simulate latency between 20ms and 600ms
        duration = random.uniform(0.02, 0.6)
        time.sleep(duration)
        status = '200' if random.random() > 0.1 else '500'
    finally:
        elapsed = time.time() - start_time
        REQUEST_LATENCY.labels(endpoint=endpoint).observe(elapsed)
        REQUEST_COUNT.labels(method=method, endpoint=endpoint, status=status).inc()
        ACTIVE_WORKERS.dec()

if __name__ == '__main__':
    start_http_server(8000)
    print("Metrics endpoint ready at: http://localhost:8000/metrics")
    
    while True:
        process_request()
        time.sleep(0.05)

2. Production Best Practices: Avoiding High Cardinality Crashes

Beware the High Cardinality Trap: Never put User IDs, email addresses, UUIDs, or Order IDs into metric labels. If your system has 100,000 users, each metric will spawn 100,000 distinct time series. Prometheus will consume all available memory and get OOMKilled in a matter of hours. Labels should only contain bounded, low-cardinality values such as HTTP status codes, API routes, service names, or HTTP methods.
Follow Standard Naming Conventions: Metric names should include base units (time, bytes) and Counters should end in _total. For example: http_request_duration_seconds (Histogram), http_requests_total (Counter), db_connection_pool_active (Gauge).
Tailor Histogram Buckets to Reality: Default Histogram buckets span from 5ms to 10s. If you are profiling Elasticsearch queries that range from 200ms to 30s, buckets below 100ms are completely redundant and waste memory.
Leverage Built-in Decorators: Instead of boilerplate try/finally blocks, use clean decorators like @REQUEST_LATENCY.labels(endpoint='/api/pay').time() or @ACTIVE_WORKERS.track_inprogress() directly on your target functions.

3. Configuring Prometheus Scrape Jobs

Create a prometheus.yml file alongside the Docker Compose configuration so Prometheus periodically pulls metrics every 5 seconds:

global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: 'python-app'
    static_configs:
      - targets: ['host.docker.internal:8000']

Verification & Visualization

1. Inspecting Raw Metrics with curl

Send a request to verify that the application renders the standard Prometheus text format correctly:

curl -s http://localhost:8000/metrics | grep -E "app_http_requests_total|app_active_workers"

The output shows metrics properly structured with labels:

# TYPE app_http_requests_total counter
app_http_requests_total{endpoint="/api/v1/orders",method="GET",status="200"} 482.0
app_http_requests_total{endpoint="/api/v1/checkout",method="POST",status="500"} 53.0
# TYPE app_active_workers gauge
app_active_workers 1.0

2. Writing Basic PromQL Queries

Open the Prometheus web UI at http://localhost:9090/graph and test the following queries:

Request Throughput (RPS):

sum(rate(app_http_requests_total[1m])) by (endpoint)

5xx Error Rate (%):

(sum(rate(app_http_requests_total{status=~"5.."}[5m])) / sum(rate(app_http_requests_total[5m]))) * 100

95th Percentile Latency (p95) by Endpoint:

histogram_quantile(0.95, sum(rate(app_request_latency_seconds_bucket[5m])) by (le, endpoint))

3. Building Dashboards in Grafana

Navigate to http://localhost:3000 and log in with default credentials: admin/admin.
Go to Connections > Data Sources, select Prometheus, set the Server URL to http://prometheus:9090, and click Save & Test.
Create a new dashboard and add three essential panels:
- Stat: Query app_active_workers, configuring threshold colors (e.g., yellow above 5 workers, red above 10).
- Time Series: Plot RPS and p95 latency graphs to detect traffic spikes and performance anomalies.
- Pie Chart or Bar Gauge: Display the distribution ratio of HTTP 200 vs 500 responses for a quick glance at API reliability.

Instrumenting custom metrics takes only about 15 minutes of coding, yet it provides debugging and incident analysis capabilities far superior to traditional log analysis.