Posted inMonitoring

Building a Custom Prometheus Exporter with Python: Comprehensive Internal API Monitoring

Monitoring tutorial - IT technology blog
Monitoring tutorial - IT technology blog

When the Dashboard is ‘Green’ but the System is ‘Down’

The PagerDuty alarm goes off at 2 AM. I jump up, rubbing my eyes to look at the Grafana Dashboard: CPU is at 20%, RAM is plenty, and Node Exporter reports everything is fine. However, the customer support group chat is exploding because 500 users cannot complete their payments.

After 30 minutes of investigation, I discovered an internal API returning 500 errors. More than 1,200 orders were stuck in a ‘Pending’ state. Prometheus was completely blind to this because it only monitored infrastructure and couldn’t touch the internal business logic.

In the past, I usually SSH’d into the server to tail -f logs or manually run SQL queries to count errors. This approach is extremely exhausting. If you don’t find a way to ‘expose’ business data to Prometheus, you’ll be spending many more sleepless nights.

Why Node Exporter Isn’t Enough

Tools like node_exporter or mysql_exporter are very powerful, but they have a limitation: They don’t understand your business logic.

Imagine a legacy Core Banking system from 10 years ago. It doesn’t have a /metrics endpoint. It only returns raw JSON or writes logs to files. Prometheus needs data in a specific text format. If the system doesn’t ‘speak’ the same language, Prometheus is helpless.

You will need a custom solution when:

  • Data resides in internal APIs that only return raw JSON.
  • You need to count orders stuck for more than 5 minutes in the Database.
  • You want to track the actual number of users currently logged into the system.

Three Common Approaches

I considered three options to solve this problem:

  1. Textfile Collector: Write a script that runs as a cronjob and exports a .prom file. This is simple but hard to manage as the system grows.
  2. Pushgateway: Push data to an intermediary station. This is suitable for short-lived jobs (batch jobs). However, if the script dies and Pushgateway doesn’t know, you’ll receive ‘stale’ data.
  3. Custom Exporter (Recommended): Build a small service (sidecar) using Python. It fetches data, converts it to Prometheus standards, and waits for Prometheus to pull it. This is the most stable method for Production environments.

Hands-on: Writing a Custom Exporter with Python

Python is the number one choice thanks to the very easy-to-use prometheus_client library. We will write an exporter that fetches data from a simulated API.

Step 1: Install Tools

Create a virtual environment and install the necessary libraries:

pip install prometheus_client requests

Step 2: Write the Source Code

Suppose the /status API returns pending_orders: 42 and active_users: 150. Here is the code to convert them:

import time
import requests
from prometheus_client import start_http_server, Gauge

# Initialize Metrics: Gauge allows values to increase or decrease
PENDING_ORDERS = Gauge('myapp_pending_orders', 'Number of pending orders')
ACTIVE_USERS = Gauge('myapp_active_users', 'Number of online users')

def fetch_metrics():
    try:
        # Simulate a real API call
        # resp = requests.get("http://api.internal/status", timeout=5)
        # data = resp.json()
        data = {"pending_orders": 42, "active_users": 150}

        # Push data to Prometheus Client
        PENDING_ORDERS.set(data['pending_orders'])
        ACTIVE_USERS.set(data['active_users'])
        print(f"Updated: {data['pending_orders']} orders")
    except Exception as e:
        print(f"Error fetching data: {e}")

if __name__ == '__main__':
    # Open port 8000 for Prometheus to scrape data
    start_http_server(8000)
    while True:
        fetch_metrics()
        time.sleep(15) # Update every 15 seconds

Step 3: Configure Prometheus

Add the following lines to your prometheus.yml file to connect:

scrape_configs:
  - job_name: 'python_exporter'
    static_configs:
      - targets: ['localhost:8000']

After restarting Prometheus, you can immediately start creating charts in Grafana.

Hard-won Lessons from Deployment

To ensure the exporter doesn’t crash the system, keep these 3 points in mind:

  • Always use Timeouts: When calling APIs with requests, set timeout=5. If the API hangs, your exporter will also hang.
  • Match Scrape Intervals: If Prometheus pulls data every 15 seconds, your script should also run every 15 seconds. Don’t call the API too frequently to avoid wasting resources.
  • Manage with Systemd: Don’t run the script manually. Use Systemd so it automatically restarts if the server reboots or the script crashes.

Example Systemd file (/etc/systemd/system/my_exporter.service):

[Service]
ExecStart=/usr/bin/python3 /opt/exporter.py
Restart=always
User=prometheus

Now, instead of staying up all night, I just set up alerts on Grafana. If myapp_pending_orders > 100 for 2 minutes, Telegram notifies me immediately. You can handle incidents remotely without even opening your laptop. That is the true value of custom-tailored monitoring.

Share:
Tags: