Monitoring Jenkins Pipelines with Prometheus: From ‘Black Box’ to 5-Star Dashboard

Why You Shouldn’t Let Jenkins Run ‘Unchecked’

Once, my team faced a frustrating situation: our CI/CD pipeline suddenly slowed to a crawl. A typical build that usually took 10 minutes suddenly spiked to 45 minutes. The whole team scrambled to check logs on every node and inspect every commit, but found nothing unusual. That’s when I realized that Jenkins without monitoring is nothing more than a mysterious “black box.”

As systems scale to dozens of slave nodes and hundreds of overlapping jobs, manual checking becomes impossible. Prometheus combined with the Jenkins Prometheus Exporter is the ultimate lifesaver. This duo helps you shine a light on hidden corners such as:

• How many jobs are stuck in the queue (Queue time)?
• Which node is hogging RAM, causing builds to hang?
• Which projects have abnormally high failure rates in the last 24 hours?

Setting up monitoring early allows you to handle issues proactively before management even has a chance to complain. Most importantly, it eliminates the “reactive troubleshooting” cycle whenever the system hits a bottleneck.

Installing Jenkins Prometheus Exporter

Instead of installing a separate agent, we will use the Prometheus metrics plugin. This plugin is extremely lightweight and automatically converts Jenkins’ internal data into a format that Prometheus can understand.

Step 1: Install the Plugin

First, navigate to Manage Jenkins > Plugins. In the Available plugins tab, search for the keyword: Prometheus metrics. After clicking install, I recommend restarting Jenkins to ensure the plugin initializes all necessary classes.

Step 2: Check the Metrics Endpoint

Once successfully installed, Jenkins will expose a data endpoint at: http://<your-jenkins-url>:8080/prometheus/. Try accessing this link. If the screen displays a series of text lines like jenkins_builds_duration_milliseconds_summary, congratulations, you’re on the right track.

Security Tip: If you encounter a 403 Forbidden error, go to Manage Jenkins > System. In the Prometheus configuration section, you can check Path authentication disabled. However, in a Production environment, it’s best to use an API Token for Prometheus to authenticate when scraping data.

Configuring Prometheus to Scrape Data

Now it’s time to connect the two systems. Open your prometheus.yml file and add the following configuration to the scrape_configs section:

scrape_configs:
  - job_name: 'jenkins-ci'
    metrics_path: '/prometheus/'
    scrape_interval: 15s # Collect data every 15 seconds
    static_configs:
      - targets: ['192.168.1.50:8080']
    basic_auth:
      username: 'monitor-user'
      password: 'your-api-token-here'

To apply the configuration without restarting the service, run the following command:

curl -X POST http://localhost:9090/-/reload

Note: You should create a dedicated User in Jenkins with only View permissions for Prometheus. Using an Admin account to scrape data is a serious security vulnerability that many people overlook.

3 “Golden” Metrics for Your Dashboard

Once the data starts flowing in, don’t get bogged down in creating too many charts. Focus on these 3 core metrics:

1. Build Agent Health

Use the metric jenkins_node_online_status. A value of 1 is stable, 0 means disconnected. I usually set an alert if a critical build node (such as a Docker or iOS runner) stays offline for more than 3 minutes.

2. Queue Size

Use the metric jenkins_queue_size_value. If this number is consistently greater than 5 for 15 minutes, your team is severely lacking Executors. This is the time to upgrade the CPU or add more slave nodes.

3. Success Rate

The metric jenkins_builds_last_build_result_ordinal helps you track code health. A project with a continuous failure rate is often due to an unstable build environment or weak unit tests.

Connecting Grafana for Visualization

Instead of designing your own, take advantage of existing dashboards. Use ID 9964 in the Grafana Import section. This dashboard provides everything from average build time charts to current executor status in just a few seconds of setup.

Lessons in Handling “Alert Fatigue”

When I first started, I used to receive 100 Telegram messages a night just because a flickering network caused Prometheus to lose connection temporarily. That was a classic mistake. To avoid “alert fatigue,” you should:

• Use the avg_over_time function to filter out instantaneous data spikes.
• Only trigger alerts when the issue persists (e.g., Queue > 10 for 10 consecutive minutes).
• Categorize notifications: Slack alerts for minor issues, phone calls/SMS for total system failures.

Monitoring Jenkins isn’t just about installing tools; it’s about understanding how your system operates. If you have trouble writing PromQL queries, leave a comment below and I’ll help you out!