Monitoring Proxmox VE with Prometheus and Proxmox VE Exporter: Managing VMs, LXCs, and Storage

Comparing 3 Monitoring Approaches for Proxmox VE Clusters

Managing a Proxmox cluster with 3 or more nodes running 40–50 VMs/LXCs requires continuous observability. You need instant visibility into both hardware and virtual machine health. Currently, there are three common approaches:

• Option 1: Install Node Exporter on each VM/LXC. Each virtual guest runs its own process, and Prometheus scrapes metrics from each individual IP address.
• Option 2: Use Proxmox’s built-in Metric Server to push data to InfluxDB or Graphite. You simply configure it directly via the web GUI, and Proxmox periodically sends metrics automatically.
• Option 3: Deploy Proxmox VE Exporter via the REST API with Prometheus. The exporter acts as an intermediary, using an API Token to scrape metrics across the entire cluster—from physical nodes and VMs/LXCs to ZFS/Ceph storage pools.

Real-World Pros and Cons of Each Solution

1. Installing Node Exporter on Every Virtual Guest (Per-VM Agent)

This approach delivers the deepest operating system-level metrics from guest machines, giving you clear visibility into individual processes, RAM caching, and swap usage.

However, operational overhead is high. Every time a new VM is provisioned, you must install the agent, configure firewall rules, and update Prometheus scrape targets. More importantly, in-guest agents cannot alert you to physical disk failures, degraded Ceph pools, or unresponsive KVM host processes.

2. Built-in Metric Server (Pushing to InfluxDB)

Setup is lightning fast with just a few clicks in the Proxmox GUI, requiring no additional helper services.

The downside arises if your overall monitoring infrastructure is already standardized on Prometheus and Alertmanager. Standing up a dedicated InfluxDB instance solely to ingest Proxmox metrics wastes resources and significantly complicates centralized alerting.

3. Proxmox VE Exporter Connected via REST API

This is the most practical choice for operations teams. You only need to provision a read-only API Token. The exporter automatically gathers metrics from all newly created VMs/LXCs without ever touching the guest OS. It comprehensively covers physical node hardware, allocated resources, and centralized storage pools.

Why Proxmox VE Exporter Is Ideal for Production Environments

As infrastructure scales, relying on development teams to install agents on every guest often leads to gaps. Proxmox VE Exporter cleanly decouples the virtualization layer from the application layer, giving you instant infrastructure-wide visibility independent of internal VM configurations.

Step-Step Implementation: Proxmox VE Exporter and Prometheus

Step 1: Create a User and API Token with Least Privilege

Never use the root account for monitoring. Instead, create a dedicated user under the pve realm with read-only permissions.

Open a shell on any Proxmox node and run the following commands in sequence:

# 1. Create a role with read-only access to infrastructure metrics
pveum role add MonitoringRole -privs "VM.Audit Datastore.Audit Sys.Audit Sys.Console"

# 2. Create a dedicated user
pveum user add prometheus-exporter@pve -comment "Prometheus Monitoring User"

# 3. Grant cluster-wide permissions
pveum acl modify / -user prometheus-exporter@pve -role MonitoringRole

# 4. Generate API Token (disable privilege separation to inherit user permissions)
pveum user token add prometheus-exporter@pve monitoring-token --privsep 0

The terminal will display the Value (token secret). Save this string to your password manager immediately; Proxmox will never display it again.

Step 2: Deploy Proxmox VE Exporter via Docker Compose

You can run the exporter directly on your monitoring host. Create a docker-compose.yml file as follows:

services:
  proxmox-exporter:
    image: prompve/prometheus-pve-exporter:latest
    container_name: proxmox-pve-exporter
    restart: unless-stopped
    ports:
      - "9221:9221"
    volumes:
      - ./pve.yml:/etc/prometheus/pve.yml:ro

Next, create the authentication configuration file pve.yml in the same directory:

default:
  user: prometheus-exporter@pve
  token_name: monitoring-token
  token_value: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  verify_ssl: false

Start the container and verify the connection:

docker compose up -d

# Test the exporter against a Proxmox node IP (e.g., 192.168.1.10)
curl -s "http://localhost:9221/pve?target=192.168.1.10" | grep pve_up

If the command returns pve_up 1, the exporter has successfully connected to the Proxmox API.

Step 3: Configure Prometheus Scrape Targets

Proxmox VE Exporter functions via a query proxy pattern similar to the Blackbox Exporter. Open prometheus.yml and add the following configuration:

scrape_configs:
  - job_name: 'proxmox-ve'
    scrape_interval: 20s
    scrape_timeout: 15s
    metrics_path: /pve
    static_configs:
      - targets:
          - '192.168.1.10:8006'  # Node pve-01 IP
          - '192.168.1.11:8006'  # Node pve-02 IP
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: '192.168.1.200:9221'  # Exporter host IP

Reload the Prometheus configuration:

curl -X POST http://localhost:9090/-/reload

Production Insights & Best Practices for Proxmox Monitoring

1. Essential PromQL Queries for Your Dashboard

• Actual VM memory usage percentage: Requires qemu-guest-agent installed inside the guest for accurate RAM metrics.

  (pve_guest_memory_used_bytes / pve_guest_memory_size_bytes) * 100

• Detect unexpected VM shutdown or crashes:

  pve_guest_info{type="qemu"} == 0

• Alert when storage exceeds the safe threshold (above 85%):

  (pve_storage_used_bytes / pve_storage_size_bytes) * 100 > 85

2. Fine-Tuning the Scrape Interval

Each scrape cycle triggers multiple API queries against the cluster. In an environment with over 50 VMs, setting scrape_interval: 5s will cause the pvedaemon process to consume sustained high CPU. Maintain an interval between 15s and 30s instead—this provides sufficient granularity for incident detection without overwhelming host APIs.

3. Always Monitor Disk I/O Wait

CPU is rarely the primary bottleneck in virtualization systems; disk performance usually is. When multiple VMs write data concurrently, the physical node’s iowait can spike dramatically. Closely monitoring pve_storage_* metrics and I/O delay helps you detect degrading disks or congested ZFS pools well before the entire cluster locks up.