How to Monitor Harbor Container Registry with Prometheus and Grafana

Monitoring tutorial - IT technology blog
Monitoring tutorial - IT technology blog

The 2:00 AM Nightmare: When Harbor Runs Out of Disk Space

At exactly 2:15 AM, my phone started ringing off the hook. The on-call team was frantically pinging me with blood-red screenshots from the CI/CD pipeline: Error: write /var/lib/docker/tmp/docker-builder...: no space left on device. Cascading right after, nearly 40 deployments across the production Kubernetes cluster were stuck in ImagePullBackOff.

I rushed to open the Harbor admin dashboard. The page spun endlessly. The 2TB disk storing artifacts had hit 99.8% capacity. Worse yet, the replication process to the DR backup cluster had quietly hung for six hours without triggering a single alert on any channel.

Running a registry for over 60 engineers without metrics is like driving blind at night. Waiting for a major outage before digging through raw text logs is already too late.

How Harbor Exposes Metrics

Starting with version 2.2, Harbor includes a built-in native Prometheus endpoint. You do not need to install an external exporter.

The system categorizes metrics into two primary groups:

  • Harbor Core Metrics: Reports project quota status, actual storage usage, artifact counts, and replication task progress.
  • Registry Metrics: Tracks HTTP status codes (200, 404, 500), latency, and the frequency of docker push and docker pull commands.

Harbor aggregates all this information onto a single port (defaulting to 9090) at the /metrics path. Prometheus periodically scrapes this data into its TSDB, while Grafana queries it to render dashboards.

Step-by-Step Integration with Prometheus & Grafana

Step 1: Enable the Metrics Endpoint in harbor.yml

Access the server hosting Harbor and navigate to the configuration directory:

cd /opt/harbor
sudo vim harbor.yml

Locate the metric block and update it as shown below:

metric:
  enabled: true
  port: 9090
  path: /metrics

Save the file and apply the updated container configuration:

sudo ./prepare
sudo docker compose down
sudo docker compose up -d

Run a test curl command on localhost to verify the exporter:

curl -s http://127.0.0.1:9090/metrics | head -n 20

If the terminal returns a list prefixed with # HELP harbor_..., the endpoint is up and running.

Step 2: Define the Scrape Target in Prometheus

Open the prometheus.yml file on your monitoring server and add a new job:

scrape_configs:
  - job_name: 'harbor-monitoring'
    scrape_interval: 15s
    scrape_timeout: 10s
    static_configs:
      - targets: ['192.168.10.20:9090']
        labels:
          env: 'production'
          service: 'registry-harbor'

Reload Prometheus without interrupting the service:

curl -X POST http://localhost:9090/-/reload

Navigate to Status > Targets in the Prometheus web UI. Once the harbor-monitoring target turns green with an UP state, you are all set.

Step 3: Set Up Essential PromQL Queries

Here are 3 must-have queries to include on your Grafana Dashboard.

1. Storage Quota Usage by Project

Prevent scenarios where a single repository suddenly bloats and crashes the entire cluster. Query the percentage of storage used:

# Percentage of storage used per project
(harbor_project_quota_used_bytes / harbor_project_quota_total_bytes) * 100

If projects do not have fixed quotas assigned, track the actual storage consumption in gigabytes (GB) instead:

# Storage consumption converted to GB
harbor_project_quota_used_bytes{project_name="core-banking"} / 1024 / 1024 / 1024

2. Image Pull and Push Frequency

This metric provides early detection of CI/CD crash loops or a worker node trapped in an infinite pull loop:

# Image pull rate (req/s)
sum(rate(harbor_registry_http_requests_total{handler="/v2/{name:.*}/manifests/{reference:.*}", method="GET"}[5m]))

# New image push rate (req/s)
sum(rate(harbor_registry_http_requests_total{handler="/v2/{name:.*}/blobs/uploads/{uuid:.*}", method="PUT"}[5m]))

3. Cross-Cluster Replication Status

When synchronizing to a disaster recovery (DR) cluster, any failed task must be caught immediately:

# Number of failed replication tasks in the last 10 minutes
increase(harbor_replication_task_total{status="failed"}[10m])

Step 4: Configure Alert Rules and Filter Out Noise

Alert fatigue quickly dulls an on-call team’s reflexes. Initially, I set an 80% storage threshold with a hold duration of for: 1m. As a result, every time developers ran multi-stage builds or cleared caches, Telegram went off in the middle of the night. The team grew exhausted and gradually muted notifications.

The solution is to split the alert into two tiers with sensible hold durations:

groups:
  - name: harbor_alerts
    rules:
      - alert: HarborStorageQuotaWarning
        expr: (harbor_project_quota_used_bytes / harbor_project_quota_total_bytes) * 100 > 85
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Harbor project {{ $labels.project_name }} has reached 85% capacity"
          description: "Project storage has exceeded 85% for 30 consecutive minutes. Plan an artifact cleanup."

      - alert: HarborStorageQuotaCritical
        expr: (harbor_project_quota_used_bytes / harbor_project_quota_total_bytes) * 100 > 95
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Harbor project {{ $labels.project_name }} is almost out of disk space"
          description: "Storage has exceeded 95%. Purge old images or increase the quota immediately."

      - alert: HarborReplicationFailed
        expr: increase(harbor_replication_task_total{status="failed"}[5m]) > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Replication task failed"
          description: "Failed to replicate images to the DR cluster. Check network connectivity or authentication tokens."

Holding the 85% threshold for 30 minutes eliminates transient spikes caused by temporary builds. Conversely, triggering at 95% for 5 minutes immediately wakes up the on-call engineer when a cluster outage is imminent.

Conclusion

Don’t wait for the Docker daemon to throw no space left on device before scrambling to clean up disks. Setting up metrics for storage quotas, request rates, and replication status takes only about 20 minutes of configuration. In return, you gain complete visibility into storage health and keep your CI/CD pipelines running smoothly.

Share: