Grafana Mimir: An ‘Infinite’ Prometheus Metrics Storage Solution on Object Storage

The ‘Disk Full’ Anxiety When Running Prometheus

If you’re operating a Prometheus cluster monitoring about 20-30 servers, you’ve likely received ‘disk full’ alerts in the middle of the night. By default, Prometheus stores data on local disks. As metrics grow or when you need to retain data for 1-2 years for reporting, SSD/NVMe costs become prohibitively expensive. Manually backing up and moving hundreds of GBs of TSDB data is also a nightmare.

My system previously only dared to keep data for 15 days to protect the disk. After deploying Mimir, I was able to offload all old metrics to Object Storage (S3/MinIO). This allows retrieving trends from 6 months ago in seconds without spending an extra cent on physical disks. Mimir is an open-source project from Grafana Labs, capable of handling billions of metrics with an extremely flexible architecture.

Why I Chose Mimir Over Thanos?

Many of you often consider Thanos because of its popularity. However, after trying both, I found Mimir offers more practical advantages:

• Lightning-fast deployment: Mimir supports monolithic mode. You only need a single binary to run the entire system, instead of managing 5-7 discrete components like Thanos.
• Push instead of Pull mechanism: Mimir receives data via remote_write. You don’t need to install additional Sidecars alongside Prometheus, reducing the load on monitoring nodes.
• Query speed: Thanks to smart sharding and hashing mechanisms, Mimir processes complex queries on large datasets significantly faster.

Operational Model

The data flow works like this: Prometheus collects metrics and immediately pushes them to Mimir via remote_write. Here, Mimir compresses the data and stores it long-term in MinIO (Object Storage). Finally, Grafana connects directly to Mimir to fetch data for visualization.

Installation Guide with Docker Compose

To get started, you need a Linux server with Docker installed. We will build a complete lab cluster including MinIO, Mimir, Prometheus, and Grafana.

1. Setting up MinIO (Metrics Storage)

Create a docker-compose.yml file. MinIO will act as a low-cost storage replacement for AWS S3.

services:
  minio:
    image: minio/minio
    container_name: mimir-minio
    volumes:
      - ./minio-data:/data
    environment:
      MINIO_ROOT_USER: mimiradmin
      MINIO_ROOT_PASSWORD: mimirpassword
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"
      - "9001:9001"

2. Configuring Grafana Mimir

Create a mimir-config.yaml file. This is the most crucial part for connecting Mimir to MinIO.

target: all
common:
  storage:
    backend: s3
    s3:
      endpoint: minio:9000
      access_key_id: mimiradmin
      secret_access_key: mimirpassword
      insecure: true
      bucket_name: mimir-metrics

multitenancy_enabled: false

blocks_storage:
  backend: s3
  s3:
    bucket_name: mimir-blocks
  tsdb:
    dir: /tmp/mimir/tsdb

server:
  http_listen_port: 9009

3. Connecting Prometheus to Mimir

Open the prometheus.yml file and add the remote_write configuration. This command tells Prometheus to push data as soon as it is collected.

global:
  scrape_interval: 15s

remote_write:
  - url: http://mimir:9009/api/v1/push

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

4. Activating the System

Add Mimir, Prometheus, and Grafana to docker-compose.yml and run docker-compose up -d. After a few minutes, your entire monitoring stack will be ready.

Configuring Data Source in Grafana

Access Grafana at port 3000. Instead of pointing directly to Prometheus (port 9090), add a new Prometheus Data Source with the URL http://mimir:9009/prometheus.

The beauty of this is that Grafana will see Mimir as a giant Prometheus server. Mimir automatically aggregates new data from RAM and old data from MinIO to return the most accurate results.

Practical Operational Experience

After running Mimir in production for a while, I have 3 important tips for you:

• Manual Bucket Creation: Mimir does not automatically create buckets on MinIO. You must access port 9001 and create two buckets named mimir-metrics and mimir-blocks before starting Mimir.
• RAM is Critical: Mimir needs RAM to index and compress data. For 15-20 servers, allocate at least 4GB of RAM to prevent containers from restarting due to OOM (Out of Memory).
• Keep Prometheus Retention Short: You should set --storage.tsdb.retention.time=2h on Prometheus. This minimizes local disk load since Mimir handles the long-term data.

Conclusion

Switching to Grafana Mimir is the right move if you want a more professional and resilient monitoring system. You no longer have to worry about deleting old data to make room for new data every week. With the extremely low cost of Object Storage, Mimir lets you store metrics for as long as you want while ensuring stable query speeds.