Monitoring Longhorn on Kubernetes: Prometheus & Grafana Setup with Real-World Troubleshooting - ITFROMZERO - Share tobe shared! Longhorn storage clusters can degrade silently while apps keep running, risking data loss on subsequent node failures. Learn how to set up Prometheus, Grafana, and fine-tune alerts to prevent false alarms.

Table of Contents

When Your Storage Cluster Silently Fails

Deploying PostgreSQL, MySQL, or Kafka on Kubernetes often comes with the challenge of choosing the right storage. For many teams, Longhorn is a top choice. This distributed block storage solution is easy to install via Helm, offers an intuitive management UI, and automatically rebuilds replicas whenever a node goes down.

However, distributed storage comes with a major hidden danger: it often fails silently. When a pod crashes, Lens or k9s immediately flashes red. But when a Longhorn volume becomes degraded due to a lost replica? Your application continues reading and writing normally. You assume everything is fine—until a second node encounters an issue, and the volume instantly falls into a Faulted state. At that point, the database goes down, resulting in total data downtime.

Another common risk is disk I/O bottlenecks. Write latency can spike from 3ms to 250ms, choking database operations while the node’s CPU and RAM metrics remain completely green. To ensure stable operations, you need comprehensive monitoring for Longhorn as soon as your cluster is set up.

How Longhorn Exposes Metrics

The Longhorn manager runs as a DaemonSet across every node. It exposes a standard Prometheus /metrics endpoint on port 9500. Here are the 4 most critical metric categories you should keep on your radar:

Volume Robustness: The longhorn_volume_robustness metric reflects volume health. A value of 1 indicates Healthy (all replicas intact), 2 means Degraded (missing a replica, rebuilding in progress), and 3 represents Faulted (completely failed, cannot be mounted).
Replica State: Use longhorn_replica_state in conjunction with longhorn_node_storage_capacity_bytes to identify replicas disconnected due to disk exhaustion or internal network issues.
Read/Write Performance (IOPS & Throughput): Metrics such as longhorn_volume_read_ops, longhorn_volume_write_ops, and byte rates help pinpoint workloads that are thrashing physical disks.
Over-provisioning Ratio: By default, Longhorn allows over-provisioning up to 200%. If your cluster has 1TB of physical SSD storage, you can provision up to 2TB in PVCs. If services write data simultaneously past 1TB, the entire cluster will run out of disk space before OS-level alerts even trigger.

Hands-on: Configuring Longhorn Monitoring with Prometheus Operator

If your cluster already has kube-prometheus-stack installed, follow these 3 steps to automatically scrape metrics and configure alerting.

Step 1: Create a ServiceMonitor for Longhorn Manager

Create a longhorn-servicemonitor.yaml file to configure Prometheus to scrape metrics from the longhorn-system namespace every 30 seconds:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: longhorn-prometheus-servicemonitor
  namespace: longhorn-system
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: longhorn-manager
  namespaceSelector:
    matchNames:
      - longhorn-system
  endpoints:
  - port: manager
    path: /metrics
    interval: 30s
    scrapeTimeout: 10s

Apply the configuration:

kubectl apply -f longhorn-servicemonitor.yaml

Wait about a minute, then open the Prometheus UI and search for longhorn_volume_actual_size_bytes. If the query returns graph data, your scrape pipeline is working as expected.

Step 2: Install the Grafana Dashboard

You don’t need to build a dashboard from scratch. The Longhorn community provides comprehensive, ready-to-use templates.

Navigate to Grafana and go to Dashboards > New > Import.
Enter Dashboard ID: 13032 (Official Longhorn Overview) or 13437, then click Load.
Select your cluster’s Prometheus data source and click Import.

This dashboard provides an instant overview: actual storage usage versus committed capacity, real-time read/write IOPS per volume, and a list of unhealthy replicas.

Step 3: Configure Essential Alerting Rules

Dashboards only help when you are actively looking. Automated alerts are what save your system at 2 AM. Create a longhorn-alerts.yaml file:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: longhorn-alerts
  namespace: longhorn-system
  labels:
    release: prometheus
spec:
  groups:
  - name: longhorn.rules
    rules:
    - alert: LonghornVolumeDegraded
      expr: longhorn_volume_robustness == 2
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Volume {{ $labels.volume }} is degraded"
        description: "Volume {{ $labels.volume }} has been missing at least 1 replica for more than 10 minutes. Check the underlying disk or storage node."

    - alert: LonghornVolumeFaulted
      expr: longhorn_volume_robustness == 3
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "Volume {{ $labels.volume }} is FAULTED!"
        description: "Volume has lost all replicas or cannot be mounted. Data is currently unavailable!"

    - alert: LonghornDiskUsageHigh
      expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) * 100 > 85
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Disk usage on node {{ $labels.node }} exceeds 85%"
        description: "Longhorn storage on node {{ $labels.node }} has reached {{ $value | printf `%.1f` }}%. Add more disk space or clean up snapshots."

Deploy the alerting rules to the cluster:

kubectl apply -f longhorn-alerts.yaml

Battle-Tested Tips: Tuning Alerts to Prevent False Alarms

1. Minimize Alert Fatigue from Node Restarts

When starting out, many engineers configure the longhorn_volume_robustness == 2 alert to fire after just 1 minute. Consequently, during rolling updates or worker node restarts for kernel patches, Longhorn needs 3 to 7 minutes to resynchronize a 50GB volume. Slack and Telegram channels end up bombarded with false alarms.

The solution is to increase the for: duration to 10m or 15m for warning-level alerts. If resynchronization has not completed after 10 minutes, you are likely dealing with a genuine hardware or infrastructure failure: bad sectors, network partitioning, or a read-only filesystem.

2. Track Down Noisy Neighbors Using IOPS Metrics

When API response times suddenly spike, don’t just look at CPU utilization. Check the IOPS panel in Grafana and run the following query:

topk(5, rate(longhorn_volume_read_ops[5m]) + rate(longhorn_volume_write_ops[5m]))

This query immediately isolates the top 5 I/O-heavy volumes. In practice, a single unindexed reporting query running a full table scan in MySQL can easily sustain 3,500 IOPS—enough to choke other containers sharing a SATA SSD (capped around 500–600 IOPS) on the same node.

3. Prevent “Zombie” Snapshots

Longhorn’s recurring snapshot feature is very useful, but without an adequate retention policy, snapshot data will quietly consume all available disk space. You should plot a comparison between longhorn_volume_actual_size_bytes and the requested capacity in your PersistentVolumeClaim (PVC).

If a 20GB volume consumes 60GB of actual disk space, you likely have dozens of unpurged legacy snapshots. Set up a recurring CronJob for snapshot purging or configure a sensible Retain count (e.g., keeping only the last 3–5 snapshots) to reclaim unused storage blocks.

Summary

Operating storage on Kubernetes leaves no room for guesswork. By combining a ServiceMonitor, Grafana Dashboard ID 13032, and sensible alert thresholds, you can maintain full visibility into read/write performance, actual disk usage, and replica health. Test this setup in staging, simulate a node failure, and observe Longhorn’s rebuilding behavior firsthand before promoting to production.