Prometheus Federation: Aggregating Metrics from Multiple Clusters into a Single Global View – ITFROMZERO

Table of Contents

When Your Infrastructure Grows Beyond Expectations

I started with a single Prometheus instance — enough to monitor 5–6 production servers, and everything ran smoothly. But when the company expanded with a Kubernetes cluster in Singapore, then added a data center in Tokyo, each location running its own Prometheus, the question hit immediately: how do you view the entire infrastructure on a single Grafana dashboard?

Before centralized monitoring, I had to SSH into each server to check its status — getting a 2 AM alert saying “Singapore cluster has issues” meant opening 3 terminals at once and connecting to 3 different locations to track down the root cause. That’s an experience nobody wants to repeat. Now a single dashboard shows everything — but to get there, you need to solve the problem of aggregating metrics from multiple Prometheus instances into a single point.

Why Not Simply Connect Them Directly?

Prometheus operates on a pull-based model: it actively scrapes metrics from targets within the same network. This means each Prometheus instance can only “see” servers that are close to it on the network. Prometheus in Singapore can’t directly scrape pods in the Tokyo cluster — not just because of latency, but because VPC isolation, firewalls, and security policies between cloud regions simply don’t allow it.

The result is multiple Prometheus instances running in parallel, each an isolated “island”:

Singapore cluster: Prometheus A — knows Singapore’s metrics
Tokyo cluster: Prometheus B — knows Tokyo’s metrics
Hanoi data center: Prometheus C — knows the Hanoi DC’s metrics

Grafana can connect to multiple data sources, but if you want to write a single PromQL query to calculate total CPU usage across your entire infrastructure, that’s impossible when each Prometheus has no knowledge of the others’ existence.

Common Approaches to This Problem

Connecting Multiple Data Sources in Grafana

The simplest approach: add each Prometheus to Grafana as a separate data source and use the “Mixed” datasource feature to plot multiple metrics on the same panel. This has clear limitations — you can’t use a single PromQL query to aggregate data from multiple sources, and setting up alerts becomes fragmented and difficult to manage.

Remote Write to Centralized Storage

Prometheus supports remote_write to push all metrics to a centralized storage system like Thanos, Cortex, or VictoriaMetrics. This is a powerful solution for large-scale production, but the overhead is significant — you need to deploy an additional complex system, incurring extra cost and operational effort. You don’t always need a “bazooka” for a reasonably-sized problem.

Prometheus Federation

The “middle ground” solution built directly into Prometheus, requiring no additional components. You set up a Global Prometheus (“parent”) that scrapes pre-aggregated metrics from local Prometheus instances (“children”) via the /federate endpoint.

Prometheus Federation — The Best Approach for Small to Mid-Scale Infrastructure

Federation is the right fit when you have 2–10 clusters/regions, don’t want to deal with Thanos or Cortex, and need queries at the “global overview” level — you don’t need to drill down to specific pods from the Global Prometheus (you still do that directly on each child).

How It Works

Each child Prometheus exposes the /federate endpoint. The Global Prometheus scrapes this endpoint, pulling back a pre-filtered set of metrics. The child retains all detailed metrics; the global only pulls what’s needed for overview and global alerting.

[Child Prometheus - Singapore] ──┐
[Child Prometheus - Tokyo]    ──┼──▶ /federate ──▶ [Global Prometheus] ──▶ [Grafana]
[Child Prometheus - Hanoi]    ──┘

Verifying the /federate Endpoint on Child Instances

First, confirm that the child Prometheus is exposing data via the /federate endpoint:

# Check the /federate endpoint — should return text/plain data
curl 'http://prometheus-singapore:9090/federate?match[]={job="node_exporter"}'

# To view all available metrics
curl 'http://prometheus-singapore:9090/federate?match[]={__name__=~".+"}'

Configuring Global Prometheus (prometheus.yml)

This is the core configuration. The Global Prometheus needs a special scrape job for each child. Note that honor_labels: true is mandatory — without it, the Global will overwrite the instance and job labels with its own values, losing the metric’s origin information.

global:
  scrape_interval: 60s      # Global scrapes slower than local (15s) — used for overview
  evaluation_interval: 60s

scrape_configs:
  # Federation from Singapore cluster
  - job_name: 'federate-singapore'
    scrape_interval: 60s
    honor_labels: true        # Preserve labels from child — REQUIRED
    metrics_path: '/federate'
    params:
      match[]:
        - '{job="node_exporter"}'            # CPU/RAM/Disk for all nodes
        - '{job="kube-state-metrics"}'       # Kubernetes cluster state
        - 'up'                               # Health check for all targets
        - '{__name__=~"instance:.*"}'        # Pre-aggregated recording rules
    static_configs:
      - targets:
          - 'prometheus-singapore.internal:9090'
        labels:
          cluster: 'singapore'
          region: 'ap-southeast-1'

  # Federation from Tokyo cluster
  - job_name: 'federate-tokyo'
    scrape_interval: 60s
    honor_labels: true
    metrics_path: '/federate'
    params:
      match[]:
        - '{job="node_exporter"}'
        - '{job="kube-state-metrics"}'
        - 'up'
        - '{__name__=~"instance:.*"}'
    static_configs:
      - targets:
          - 'prometheus-tokyo.internal:9090'
        labels:
          cluster: 'tokyo'
          region: 'ap-northeast-1'

  # Federation from Hanoi DC (bare-metal, no Kubernetes)
  - job_name: 'federate-hanoi'
    scrape_interval: 60s
    honor_labels: true
    metrics_path: '/federate'
    params:
      match[]:
        - '{job="node_exporter"}'
        - 'up'
    static_configs:
      - targets:
          - '10.0.1.50:9090'
        labels:
          cluster: 'hanoi'
          region: 'ap-southeast-7'

Optimizing with Recording Rules on Child Prometheus

Instead of federating raw metrics (which wastes bandwidth and increases cardinality), create recording rules on each child to pre-aggregate the data. The Global Prometheus only needs to pull precomputed results — far more efficient and compact.

# On Child Prometheus: rules/federation-aggregates.yml
groups:
  - name: federation_aggregates
    interval: 30s
    rules:
      # Average CPU usage per node
      - record: instance:node_cpu_usage:rate5m
        expr: |
          1 - avg by (instance) (
            rate(node_cpu_seconds_total{mode="idle"}[5m])
          )

      # Memory usage percentage
      - record: instance:node_memory_usage:ratio
        expr: |
          1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

      # Disk usage percentage (mount point /)
      - record: instance:node_disk_usage:ratio
        expr: |
          1 - (
            node_filesystem_avail_bytes{mountpoint="/"}
            / node_filesystem_size_bytes{mountpoint="/"}
          )

With recording rules in place, the Global Prometheus only needs to federate {__name__=~"instance:.*"} instead of pulling all raw metrics.

Querying the Global Prometheus After Setup

With Grafana connected to the Global Prometheus, you can write PromQL queries to get a complete infrastructure overview:

# Average CPU usage by cluster
avg by (cluster) (instance:node_cpu_usage:rate5m)

# Total number of "up" nodes by region
count by (region) (up{job="node_exporter"} == 1)

# Top 5 nodes with highest memory usage across the entire infrastructure
topk(5, instance:node_memory_usage:ratio)

# Alert: which nodes are using more than 85% disk
instance:node_disk_usage:ratio > 0.85

Things to Keep in Mind During Deployment

A 60s scrape interval is sufficient for global: Federation isn’t for real-time detail — leave that to the children. The global is for trends and overview; 60s is perfectly reasonable.
Control cardinality: Don’t federate all metrics with match[]={__name__=~".+"}. Your Global Prometheus will become just as heavy as the locals. Only pull what you actually need at the global level.
Network connectivity between clusters: The Global needs to reach each child via HTTP(S). In multi-region cloud environments, consider using VPN or private peering — don’t expose Prometheus to the public internet without authentication.
Shorter retention for Global: The Global Prometheus typically retains 7–15 days since it’s an overview layer. Each child retains detailed data for longer (30–90 days).
Basic auth if you need to secure /federate: Add basic_auth to the Global’s scrape config if the child Prometheus sits behind an authenticated reverse proxy.

Federation vs. Thanos — When Should You Switch?

Federation is sufficient when you have fewer than 10 clusters and don’t need to query far back into the past (long-term storage). When the infrastructure grows larger — dozens of clusters, months of retention needed, deduplication required for HA — that’s when to consider Thanos or VictoriaMetrics. But don’t rush to Thanos when 3 clusters are perfectly manageable with Federation: the added complexity simply isn’t worth it for the problem at hand.

With my current setup — 3 regions, Federation running stably — the Grafana dashboard shows the full infrastructure status within under 2 minutes of an incident occurring. Compared to the days of SSH-ing into each server to investigate, this is a meaningful quality-of-life improvement for the entire team.