Monitoring Java Apps with Prometheus & Grafana: Don’t Wait for OutOfMemory to Worry – ITFROMZERO

Table of Contents

2 AM Alerts and Late Nights “Digging” Through Logs

When I first started operating Java microservices clusters, my biggest fear was receiving a system crash notification in the middle of the night. Back then, the first thing I would do was SSH into the server and frantically type top -H, jstat, or jstack to diagnose the issue. It felt like trying to fix a car in the dark without a flashlight—extremely reactive and exhausting.

After implementing the trio of Prometheus JMX Exporter, Prometheus, and Grafana, everything changed. Instead of waiting for the app to crash, I can now just look at the Dashboard to know exactly when an incident is about to occur. This article distills practical experience to help you build a professional monitoring system for Java applications.

JMX Exporter: The “Translator” for JVM

Every Java Virtual Machine (JVM) comes with JMX for resource management. However, Prometheus cannot read this data directly due to format differences. JMX Exporter acts as a javaagent, running alongside the application to convert metrics into HTTP/text format that Prometheus understands.

Why choose an agent over running a separate service? Through many projects, I’ve found the agent to be much more stable. It starts with the app, requires no additional external process management, and minimizes latency when scraping data.

Practical Implementation Steps

Step 1: Download the JMX Exporter Agent

You need the agent’s JAR file to embed into your app. Download the latest version (e.g., 0.20.0) from Maven Central or GitHub. This file is quite lightweight, only a few MBs.

mkdir -p /opt/monitoring
cd /opt/monitoring
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.20.0/jmx_prometheus_javaagent-0.20.0.jar

Step 2: Smart config.yaml Configuration

If you don’t filter metrics, JMX will push thousands of redundant parameters, bloating Prometheus. Here is the filter I often use to focus on the most critical indicators like CPU, Threads, and GC:

# /opt/monitoring/config.yaml
startDelaySeconds: 0
ssl: false
lowercaseOutputName: true
rules:
  - pattern: 'java.lang<type=OperatingSystem><>((?!processCpuTime)\w+):'
    name: os_$1
    type: GAUGE
  - pattern: 'java.lang<type=Threading><>ThreadCount:'
    name: jvm_threads_current
    type: GAUGE
  - pattern: 'java.lang<type=Memory><>HeapMemoryUsage:(.*):'
    name: jvm_memory_heap_$1
    type: GAUGE
  - pattern: 'java.lang<type=GarbageCollector, name=(.*)><>CollectionCount:'
    name: jvm_gc_collection_count
    labels:
      gc: "$1"
    type: COUNTER

Step 3: Attach the Agent to the Application

You just need to add a -javaagent parameter to your startup command. Note: Choose a port (e.g., 8080) that does not conflict with your main Web App port.

java -javaagent:/opt/monitoring/jmx_prometheus_javaagent-0.20.0.jar=8080:/opt/monitoring/config.yaml \
     -jar your-app.jar

After running, access http://<Server-IP>:8080/metrics. If you see text lines like jvm_memory_heap_used, you have succeeded.

Step 4: Connect with Prometheus and Grafana

In the prometheus.yml file, add a job so Prometheus can periodically scrape the data. I usually set the scrape_interval to about 15s, which is sufficient for real-time monitoring.

scrape_configs:
  - job_name: 'java-microservice'
    static_configs:
      - targets: ['192.168.1.10:8080']

Finally, go to Grafana and Import Dashboard ID 8563. You will immediately have a professional monitoring interface without the effort of creating charts manually.

3 “Life-or-Death” Metrics to Watch Closely

Don’t get overwhelmed by too many charts. Based on troubleshooting experience, you only need to focus on the following three areas:

Heap Memory Usage: If the chart shows a sawtooth pattern but the troughs are getting higher, the app definitely has a Memory Leak. Set an alert when Heap exceeds 85%.
GC Stop-The-World: If jvm_gc_collection_seconds_sum spikes, the app will freeze. A GC cycle lasting over 200ms usually makes users feel a noticeable lag.
Thread Count: A continuous increase in thread count is often due to deadlocks or database connection pool saturation.

Conclusion

Monitoring isn’t just about looking at pretty charts. It helps you shift from a reactive to a proactive state. Instead of saying “I think the server is slow,” you can confidently state: “The heap is 90% full due to a memory leak in module X.” Wishing you peaceful nights with a reliable monitoring system!