Mastering MinIO Performance: How to Set Up Prometheus & Grafana to Monitor IOPS, Throughput, and Capacity

When ‘Health Check OK’ Doesn’t Mean the System is Running Fast

Have you ever encountered a situation where MinIO still shows a green status and server CPU load is only 10%, but users complain that uploading a 5MB file takes 30 seconds? I once spent hours struggling with htop and df -h without finding the cause, even though disk space was still 40% free.

The problem is that standard OS monitoring tools cannot see specific Object Storage metrics. You won’t know if disk IOPS has hit a bottleneck or if a spike in S3 API calls is filling up the queue. Without a proper dashboard, debugging is like looking for a needle in a haystack.

Why Traditional Monitoring Often Leaves You ‘Offside’

Many people have the habit of using Netdata or Zabbix to view general metrics. However, with MinIO, you need these ‘vital’ figures:

• S3 API Latency: Response time for PUT/GET commands. In a production environment, ideal latency should be under 100ms. If this number spikes to 500ms-1s, the user experience will be terrible.
• Disk Usage per Node: MinIO runs in clusters (Distributed). If just one drive in the cluster fails or responds slowly (Slow Disk), the entire write performance will be dragged down by that weakest link.
• IOPS & Throughput: Especially critical if you are using HDDs or consumer-grade SSDs. When throughput hits the 100-200MB/s limit on older drives, the system will start throwing timeout errors.
• Data Churn & Versions: If versioning is enabled, actual storage usage can double with every overwrite. You need to monitor this to avoid crashing due to sudden disk exhaustion.

Three Ways to Peek Inside MinIO

Depending on the scale of your project, you can choose the appropriate method:

1. MinIO Console: Features built-in real-time charts. However, it only displays instantaneous data. You won’t know what happened at 2 AM when the system slowed down.
2. Custom Script: Writing scripts to crawl the API and push alerts to Telegram. This is quite ‘manual,’ hard to scale, and can create noise if logs aren’t filtered carefully.
3. Prometheus + Grafana: This is the industry-standard duo in DevOps. MinIO supports exporting metrics directly in Prometheus format, helping you store months of historical data and plot accurate trend charts.

Deployment Workflow: Integrating MinIO with Prometheus & Grafana

Assuming you already have a MinIO cluster and a monitoring stack. Here are the most optimized steps to connect them.

Step 1: Configure MinIO to Export Public Metrics

By default, MinIO requires authentication signatures to fetch metrics. To simplify things for Prometheus (while still ensuring security if you block the port with a firewall), switch metrics to public mode.

Add this environment variable to your configuration file (usually /etc/default/minio or Docker Compose):

MINIO_PROMETHEUS_AUTH_TYPE="public"

After restarting, access http://<your-minio-ip>:9000/minio/v2/metrics/cluster. If you see text lines like minio_cluster_capacity_raw_total_bytes 107374182400, you have succeeded.

Step 2: Configure Prometheus to ‘Scrape’ Data

Add a new job to your prometheus.yml file. Note to set the scrape_interval to about 15s so the data isn’t too dense but still sufficient to track unusual spikes:

scrape_configs:
  - job_name: 'minio-cluster'
    metrics_path: '/minio/v2/metrics/cluster'
    static_configs:
      - targets: ['10.0.0.50:9000']

After reloading Prometheus, you can check the minio_cluster_nodes_online_total metric. If the returned value matches the number of nodes in your cluster, the connection is established.

Step 3: Use the Template Dashboard (ID 13502)

Instead of spending 4-5 hours designing charts yourself, leverage the official MinIO dashboard:

1. In Grafana, select Import.
2. Enter ID 13502.
3. Select Prometheus as the data source.

This dashboard will immediately display key metrics such as: disk status, total objects, and especially throughput charts for each individual node.

Real-world Experience: Don’t Let Alerts Ruin Your Sleep

My biggest mistake was setting an alert for when the disk reached 80% capacity. With large storage systems, going from 80% to 100% can take a month, leading to daily notifications and causing you to start ignoring them (Alert Fatigue).

My advice for you:

• Forecast instead of alarm: Use the predict_linear function in Prometheus to alert: “If this write speed continues, the disk will be full in 2 days.”
• Monitor Inodes: There are many cases where TBs of space are free, but no more files can be written because Inodes are exhausted (due to too many small files under 1KB). Don’t forget to keep an eye on the minio_node_disk_free_inodes metric.
• API Latency: Only trigger an alert if the p99 latency of PUT commands exceeds 500ms for 5 consecutive minutes.

Summary

Setting up monitoring isn’t just about having ‘pretty’ charts. It helps you shift from a reactive state (waiting for user reports) to a proactive one (knowing in advance when a disk is failing or the system is overloaded). If you are running MinIO in production, deploy the Prometheus & Grafana duo today for a better night’s sleep.