Monitoring etcd Clusters with Prometheus and Grafana: Tracking Disk Fsync Latency, Raft Proposals, and Leader Changes to Protect the Kubernetes Control Plane - ITFROMZERO - Share tobe shared! A practical guide to configuring Prometheus and Grafana for etcd cluster monitoring — focusing on the three most critical metrics: Disk Fsync Latency, Raft Proposals, and Leader Changes. Based on 6 months of production experience, including alert rules and tips to prevent etcd bottlenecks from taking down the Kubernetes Control Plane.

I had a terrible evening when my production Kubernetes cluster suddenly stopped accepting any kubectl commands. The API server was still running, nodes were still green, but every request timed out. The culprit: etcd hit a disk I/O bottleneck so severe that fsync latency spiked above 2 seconds. And I had zero alerts set up for it.

After that incident, I rebuilt the entire monitoring stack for etcd. This post is a summary of what I learned over 6 months of running it in production.

Table of Contents

Quick Start: Get a Dashboard in 5 Minutes

If you already have Prometheus + Grafana set up (see the Prometheus Operator post on this blog), etcd metrics are ready to go — you just need to configure scraping in the right place.

Step 1: Enable the Metrics Port on etcd

etcd exposes metrics at :2381/metrics by default. Check whether it’s already enabled:

# On the control plane node
curl -s http://localhost:2381/metrics | grep etcd_disk_wal_fsync

If you see output, etcd is exposing metrics. If not, add this flag to your etcd config (/etc/kubernetes/manifests/etcd.yaml with kubeadm):

spec:
  containers:
  - command:
    - etcd
    - --listen-metrics-urls=http://0.0.0.0:2381
    # ... keep other flags as-is

Step 2: Add Scrape Config to Prometheus

# prometheus-etcd-scrape.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: etcd
  namespace: monitoring
  labels:
    release: prometheus
spec:
  endpoints:
  - interval: 30s
    port: metrics
    scheme: http
    path: /metrics
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      component: etcd

kubectl apply -f prometheus-etcd-scrape.yaml

Step 3: Import the Grafana Dashboard

Dashboard ID 3070 on Grafana.com is the most standard etcd dashboard. Quick import:

# Or use the API
curl -X POST http://admin:password@localhost:3000/api/dashboards/import \
  -H 'Content-Type: application/json' \
  -d '{"dashboard":{"id":3070},"overwrite":true,"inputs":[{"name":"DS_PROMETHEUS","type":"datasource","pluginId":"prometheus","value":"Prometheus"}]}'

Go to Grafana → Dashboards → Import → enter ID 3070 → select the Prometheus datasource. Done.

The Three Most Important etcd Metrics

etcd exposes dozens of metrics, but in practice these 3 groups determine whether your cluster lives or dies:

1. Disk Fsync Latency — The #1 Cause of etcd Failures

etcd uses a WAL (Write-Ahead Log) to guarantee durability. Every write must fsync to disk before committing. When the disk is slow, the entire cluster slows down with it.

# P99 WAL fsync latency
histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (le))

# P99 Backend commit latency (bbolt)
histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (le))

Real-world thresholds:

WAL fsync < 10ms: Normal, healthy SSD
10ms – 100ms: Warning — disk is having issues
> 100ms: Critical — etcd is about to time out, Kubernetes API will become unstable

During my incident, WAL fsync climbed to 2.3 seconds because the EBS volume was being throttled after exhausting the IOPS burst limit. The lesson: etcd needs a dedicated SSD — never share it with other workloads.

2. Raft Proposals — The Heartbeat of the etcd Cluster

Raft is etcd’s consensus algorithm. Proposals are transactions proposed and committed through Raft.

# Proposal rate (transactions/second)
rate(etcd_server_proposals_committed_total[5m])

# Proposal failures — this is the dangerous metric
rate(etcd_server_proposals_failed_total[5m])

# Pending proposals (queued)
etcd_server_proposals_pending

The metric I watch most closely is proposals_pending — if this number keeps climbing without dropping, it means etcd can’t keep up with the load. This is usually caused by a disk or network bottleneck between members.

# Alert: continuous proposal failures over 5 minutes
sum(rate(etcd_server_proposals_failed_total[5m])) > 0

3. Leader Changes — Signs of an Unstable Cluster

A 3-node etcd cluster always has exactly 1 leader. A leader change happens when the current leader times out or crashes. A few leader changes per day is normal (for example during a rolling restart). But frequent leader changes are a very bad sign.

# Number of leader changes
changes(etcd_server_leader_changes_seen_total[1h])

# Whether the current node is the leader (1 = leader, 0 = follower)
etcd_server_is_leader

I once saw a cluster change leaders 50+ times in an hour because one node had >5% packet loss with the other two. The Kubernetes API didn’t go down, but latency spiked and kubectl started returning errors intermittently.

Setting Up Alerts in Practice

This is where I spent the most time. Alert fatigue was a real problem when I first set up monitoring — I initially set thresholds too low, Alertmanager fired constantly, and gradually everyone started ignoring alerts. It took a lot of tuning to find reasonable thresholds.

# etcd-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: etcd-alerts
  namespace: monitoring
spec:
  groups:
  - name: etcd.rules
    rules:
    # High WAL fsync latency
    - alert: EtcdHighFsyncLatency
      expr: |
        histogram_quantile(0.99,
          sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (le, instance)
        ) > 0.1
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "High etcd WAL fsync latency on {{ $labels.instance }}"
        description: "P99 WAL fsync = {{ $value | humanizeDuration }}, disk may be overloaded"

    # Proposal failures
    - alert: EtcdProposalsFailing
      expr: sum(rate(etcd_server_proposals_failed_total[5m])) by (instance) > 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "etcd proposals are failing on {{ $labels.instance }}"
        description: "{{ $value | humanize }} proposals/s failing — cluster may be losing quorum"

    # Too many leader changes
    - alert: EtcdHighNumberOfLeaderChanges
      expr: changes(etcd_server_leader_changes_seen_total[1h]) > 3
      labels:
        severity: warning
      annotations:
        summary: "etcd leader is changing too frequently"
        description: "{{ $value }} leader changes in 1 hour — check network and disk between nodes"

    # No leader
    - alert: EtcdNoLeader
      expr: sum(etcd_server_is_leader) == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "etcd cluster has no leader!"
        description: "Cluster has lost quorum, Kubernetes API will be unavailable"

kubectl apply -f etcd-alerts.yaml

Production Tips

Dedicate a Disk to etcd

If you’re using VMs or cloud instances, mount a dedicated volume for the etcd data directory (/var/lib/etcd). On AWS, use io2 with at least 3000 IOPS. On bare metal, use NVMe SSD. Never let etcd and containerd share a disk — container image pulls will compete for IOPS with etcd fsync operations.

Monitor Network Latency Between etcd Members

# Round-trip time between etcd peers
histogram_quantile(0.99, sum(rate(etcd_network_peer_round_trip_time_seconds_bucket[5m])) by (le, To))

If RTT between members exceeds 150ms, Raft heartbeat timeouts will start occurring. Ideally, etcd nodes should be in the same AZ or geographically close datacenters.

DB Size — An Overlooked but Critical Metric

# Actual etcd DB size
etcd_mvcc_db_total_size_in_bytes

# DB size after compaction
etcd_mvcc_db_total_size_in_use_in_bytes

etcd recommends keeping the DB under 8GB. Exceeding this significantly degrades performance. Configure auto-compaction:

# Add to etcd flags
- --auto-compaction-mode=periodic
- --auto-compaction-retention=1h

Backup Before Changing Anything

# Snapshot etcd before any maintenance
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

I run a cronjob every 6 hours and keep 7 days of backups. For a small Kubernetes cluster (<1000 resources), snapshot files are typically only 10–50MB.

When to Actually Worry

Based on real-world experience, here’s the checklist I follow when an etcd alert fires:

WAL fsync P99 > 500ms for 10+ consecutive minutes: Restart any application causing I/O contention, or scale IOPS immediately
Proposal failures > 0: First check network connectivity between all 3 etcd members
Leader changes >5 times/hour: Run etcdctl endpoint health --cluster to identify which node is unstable
DB size > 6GB: Defragment immediately: etcdctl defrag --cluster

etcd is the only component in Kubernetes where losing it means losing the entire cluster state. There’s no recovery without a backup. Good monitoring isn’t about generating lots of alerts — it’s about catching problems before they become incidents.