How to Install and Use VictoriaMetrics: An Optimized Time-Series DB to Replace Prometheus

Approaches When Prometheus Starts Getting Overloaded

In real-world monitoring, watching Prometheus swell out of control is an all-too-familiar sight. RAM gets pegged at 85–90%. A 500GB NVMe drive fills up in just three weeks of collecting metrics. And every time someone selects a 30-day time range in Grafana to check a report, the dashboard spins endlessly before timing out.

When your infrastructure hits the milestone of 2–3 million active time series, it’s decision time. Typically, you have three options:

• Vertically scale Prometheus: Upgrade server RAM from 32GB to 64GB or 128GB. Add expensive SSD storage. Meanwhile, accept cutting retention down from 30 days to a mere 7–14 days.
• Integrate Thanos or Cortex / Mimir: Offload metrics to Object Storage such as AWS S3 or MinIO. The system then requires an entire fleet of Sidecars, Store Gateways, Compactors, and Queriers.
• Migrate to VictoriaMetrics: Use VictoriaMetrics as a long-term storage backend via remote_write. If you want an even leaner setup, you can replace the scrape engine entirely with vmagent.

Detailed Comparison of the Three Scaling Options

Each approach involves clear trade-offs between hardware costs and operational overhead:

1. Pure Prometheus (Vertical Scaling)

• Pros: Easy to set up, no intermediary components, familiar configuration.
• Cons: Cannot scale horizontally. Server costs escalate quickly as metrics grow. Prometheus frequently hits OOM (Out-of-Memory) errors during data compaction or heavy queries.

2. The Thanos / Cortex / Mimir Ecosystem

• Pros: Near-infinite scalability thanks to low-cost Object Storage. Enables aggregating metrics across multiple Kubernetes clusters into a single Global Query View.
• Cons: Complex architecture with dozens of microservices. The infrastructure team must shoulder the burden of maintaining Sidecars, Stores, Queriers, and Compactors. Querying historical metrics over S3 can suffer high latency without a memcached caching layer.

3. VictoriaMetrics (Single-node & Cluster)

• Pros: Outstanding data compression, typically 5–7x better than Prometheus. Uses only about 1/4 the RAM and CPU under identical workloads. Fully compatible with PromQL syntax and the Prometheus API.
• Cons: The single-node edition is still bound to a single machine’s disk capacity. The cluster edition requires managing three separate components: vmstorage, vminsert, and vmselect.

When Should You Choose VictoriaMetrics Single-node?

If your system handles fewer than 2 million active time series and an ingestion rate under 500k samples/second, VictoriaMetrics Single-node is the top contender. You only need to run a single binary or a standalone Docker container. No MySQL/PostgreSQL required. No complex S3 bucket configurations. Operation is practically hassle-free.

Practical tip: Never rip and replace your existing system overnight. Keep your current Prometheus setup running, then enable remote_write to mirror metrics into VictoriaMetrics. Let it run for 1–2 weeks. Once you see a noticeable drop in resource usage and Grafana queries responding in under 500ms, you can safely complete the migration.

Hands-on VictoriaMetrics Deployment Guide

Step 1: Run VictoriaMetrics with Docker Compose

Deploying the service on Ubuntu via Docker Compose is the quickest and safest approach. Create a docker-compose.yml file:

version: '3.8'

services:
  victoriametrics:
    image: victoriametrics/victoria-metrics:v1.99.0
    container_name: victoriametrics
    ports:
      - "8428:8428"
    volumes:
      - vmdata:/storage-data
    command:
      - "--storageDataPath=/storage-data"
      - "--retentionPeriod=12m"
      - "--httpListenAddr=:8428"
    restart: always

volumes:
  vmdata:
    driver: local

Key parameters to note:

• --retentionPeriod=12m: Data retention period of 12 months. You can change this to 30d or 2y depending on your needs.
• --storageDataPath: Directory on disk where all time-series data is stored.
• Default listening port: 8428.

Start the container and check the service health:

docker compose up -d
curl http://localhost:8428/health

An output of OK means VictoriaMetrics is ready to ingest data.

Step 2: Configure Prometheus to Push Data to VictoriaMetrics

If you already have Prometheus scraping Node Exporters, simply add the remote_write block to your prometheus.yml file:

remote_write:
  - url: "http://10.0.0.10:8428/api/v1/write"
    queue_config:
      max_samples_per_send: 10000
      capacity: 20000
      max_shards: 30

Reload Prometheus configuration without restarting the process:

curl -X POST http://localhost:9090/-/reload

New data will be simultaneously written to VictoriaMetrics. At this point, you can safely lower Prometheus retention to 2d or 3d to instantly free up tens of gigabytes of disk space.

Step 3: Collect Metrics Without Prometheus (Using vmagent)

To drastically free up RAM, replace the Prometheus scrape worker with vmagent. Create a prometheus-scrape.yml configuration file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['10.0.0.1:9100', '10.0.0.2:9100']

  - job_name: 'victoriametrics'
    static_configs:
      - targets: ['localhost:8428']

Start the vmagent container:

docker run -d \
  --name vmagent \
  -v $(pwd)/prometheus-scrape.yml:/etc/prometheus/scrape.yml \
  victoriametrics/vmagent:v1.99.0 \
  -promscrape.config=/etc/prometheus/scrape.yml \
  -remoteWrite.url=http://victoriametrics:8428/api/v1/write

In practice, vmagent uses only about 35–50MB of RAM to scrape dozens of nodes—vastly lighter than the typical 1–2GB RAM footprint of Prometheus.

Step 4: Connect Grafana and Run Queries

VictoriaMetrics accurately emulates the Prometheus API. To connect it in Grafana:

1. Go to Grafana > Administration > Data sources > Add data source.
2. Select the Prometheus data source type.
3. In the Prometheus server URL field, enter: http://<IP_SERVER>:8428/prometheus.
4. Click Save & test. Grafana should show a green notification: Data source is working.

All your existing dashboards remain intact without any query changes. VictoriaMetrics also provides MetricsQL with numerous handy functions, such as rate_over_time, which handles intermittent data much more smoothly than standard PromQL.

Verifying Real-World Disk Compression Efficiency

The built-in web UI allows you to quickly inspect metrics and storage usage:

http://<IP_SERVER>:8428/vmui

Alternatively, you can inspect the volume directly on the Linux host:

du -sh /var/lib/docker/volumes/vmdata/_data

The compression efficiency is evident in the numbers. For a sample dataset of 100 million datapoints, Prometheus TSDB typically consumes 15 to 18GB of disk space. In comparison, VictoriaMetrics takes only about 2.5 to 3.8GB thanks to the ZSTD compression algorithm tailored specifically for float64 values and timestamps. You can keep a full year of monitoring data while directly slashing infrastructure storage costs by 60–70%.