How to Monitor HashiCorp Vault with Prometheus and Grafana: Tracking Token Leases, Request Latency, and Unseal Status - ITFROMZERO - Share tobe shared! A detailed guide to monitoring HashiCorp Vault with Prometheus and Grafana. Learn how to collect internal telemetry, measure request latency, manage token leases, and detect sealed cluster nodes early.

Table of Contents

Three Approaches to Monitoring HashiCorp Vault

In a production environment, a Vault outage means total system paralysis. Every service—from backends and workers to CI/CD pipelines—fails to retrieve database credentials or API keys. Therefore, the question arises: how should you properly monitor a Vault cluster? In practice, DevOps engineers usually consider three options:

Approach 1: Periodic health checks via HTTP endpoints — Use Prometheus Blackbox Exporter or an uptime service to continuously ping the /v1/sys/health endpoint. This method merely checks status codes such as 200 (active), 429 (standby), or 503 (sealed).
Approach 2: Aggregating audit logs to SIEM/ELK — Enable an audit device (file or syslog) and forward everything to Loki or Elasticsearch. The primary goal is to count request volumes and audit unauthorized access.
Approach 3: Enabling native telemetry via Prometheus & Grafana — Leverage Vault’s built-in telemetry engine. Cluster nodes export OpenMetrics-compliant metrics directly through the /v1/sys/metrics endpoint.

Pros and Cons of Each Approach

Each approach was designed for a specific purpose:

1. HTTP Health Check (Blackbox)

Pros: Extremely straightforward. It takes only 5 minutes of external configuration without modifying Vault’s configuration files.
Cons: Completely blind to internal metrics. You cannot tell whether Vault is handling 50,000 or 500,000 active token leases, what the write latency to Raft storage is, or if a standby node is lagging behind in replication.

2. Audit Log Analysis

Pros: Accurately records identities: which token accessed which path, from which IP, and at what time. This data is essential for compliance and security auditing.
Cons: High disk I/O and network bandwidth overhead. Most critically, by default Vault implements a fail-closed safety mechanism: if an audit device gets blocked (e.g., full disk or frozen log forwarder), Vault will block and reject all incoming requests to prevent data leakage. Turning logs into real-time alerting is a double-edged sword.

3. Native Telemetry with Prometheus

Pros: Lightweight and safe. Metrics are aggregated directly in memory and scraped periodically by Prometheus (typically every 15s or 30s). You gain visibility into every vital metric: unseal status, p99 latency distribution, expiring leases, and Raft backend snapshot health.
Cons: Requires thoughtful access control to allow Prometheus to scrape data without exposing sensitive telemetry to the public internet.

Production-Grade Architecture

Practical experience shows that the optimal architecture uses native telemetry via Prometheus + Grafana as the primary monitoring source, paired with a simple HTTP health check from outside the VPC as a network connectivity fallback.

Before setting up a centralized dashboard, our team once experienced a nerve-wracking outage: a worker node rebooted, leaving Vault sealed. This node sat idle and sealed for three days unnoticed. When the active node became overloaded and triggered a failover, our entire payment system went down because the standby node had never been unsealed. With Prometheus metrics alerting via Slack, that disaster would have been caught within the first minute.

Step-by-Step Implementation

Step 1: Enable Telemetry in HashiCorp Vault

Log in to each Vault node in your cluster (typically a 3- or 5-node Integrated Storage / Raft cluster) and open the configuration file:

sudo nano /etc/vault.d/vault.hcl

Add the telemetry stanza and enable unauthenticated metrics access within the internal network:

telemetry {
  prometheus_retention_time = "30s"
  disable_hostname          = true
}

# Listener for internal connections
listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_disable     = 0
  tls_cert_file   = "/etc/vault.d/tls/vault.crt"
  tls_key_file    = "/etc/vault.d/tls/vault.key"
  
  # Enable unauthenticated metrics (safe only within a private subnet)
  telemetry {
    unauthenticated_metrics_access = true
  }
}

Save the file and reload the Vault process without disrupting active traffic:

sudo systemctl reload vault
# Test curl directly on the node
curl -k https://127.0.0.1:8200/v1/sys/metrics?format=prometheus

If you manage multiple servers with complex configs, mixing up HCL, JSON, and Prometheus’s YAML syntax is very easy. When comparing or quickly validating syntax, I often use ToolCraft’s YAML ↔ JSON Converter and JSON Formatter. A major plus is that these tools run 100% in your browser using JavaScript, so no sensitive data is ever sent to their backend.

Step 2: Configure the Prometheus Scrape Job

Add the scrape target configuration to your prometheus.yml file. You should list all nodes in the cluster to monitor both Active and Standby instances:

scrape_configs:
  - job_name: 'hashicorp_vault'
    metrics_path: '/v1/sys/metrics'
    params:
      format: ['prometheus']
    scheme: 'https'
    tls_config:
      insecure_skip_verify: true # Change to ca_file if your setup uses a Private CA
    static_configs:
      - targets:
          - '10.0.1.10:8200'
          - '10.0.1.11:8200'
          - '10.0.1.12:8200'

Validate the configuration file syntax, then reload the Prometheus service:

promtool check config /etc/prometheus/prometheus.yml
curl -X POST http://localhost:9090/-/reload

Step 3: Build PromQL Queries and Grafana Dashboards

Open Grafana, create a new dashboard, and focus on these three critical metric groups:

1. Unseal Status and Node Role (Active/Standby)

The metric vault_core_unsealed returns 1 for healthy and 0 when sealed. Use a Stat panel with the following query:

vault_core_unsealed{job="hashicorp_vault"}

To identify which node is currently acting as the active leader handling writes, run:

vault_core_active{job="hashicorp_vault"}

In a healthy cluster, exactly one node should return a value of 1 for this metric at all times.

2. Measuring Request Latency

A slow secrets engine causes sluggish application startups and higher API response times. The query below calculates the average request handling duration over a 5-minute window:

rate(vault_core_handle_request_sum[5m]) / rate(vault_core_handle_request_count[5m])

If latency consistently spikes above 100ms (normal values are typically 5ms to 15ms), immediately inspect disk IOPS for the Raft storage volume or check for network congestion between cluster nodes.

3. Monitoring Token Leaks and Active Leases

Token leaks are the most common cause of Vault running out of memory. This happens when microservice pods continuously authenticate and generate new tokens without reusing or revoking them. Monitor these two metrics closely on a Time Series panel:

# Total active leases across the system
vault_expire_num_leases

# Total number of active tokens
vault_token_count

If you observe vault_expire_num_leases climbing in a steady staircase pattern past 100,000 without dropping, your applications are almost certainly leaking tokens or configured with excessively long TTLs.

Step 4: Set Up Alert Rules with Alertmanager

Don’t wait for an outage before checking Grafana. Set up automated alerts to notify Slack or Telegram the moment anomalies appear:

groups:
  - name: vault_alerts
    rules:
      - alert: VaultNodeSealed
        expr: vault_core_unsealed == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Vault node sealed: {{ $labels.instance }}"
          description: "Node {{ $labels.instance }} has been sealed for over 1 minute. SSH in and unseal it immediately."

      - alert: VaultHighRequestLatency
        expr: (rate(vault_core_handle_request_sum[5m]) / rate(vault_core_handle_request_count[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High Vault request latency (>100ms)"
          description: "Average response time on {{ $labels.instance }} has exceeded 100ms over the last 5 minutes."

By effectively monitoring unseal status, read/write latency, and token lease growth trends, you eliminate 95% of unexpected downtime risks for your HashiCorp Vault cluster.