The 2:00 AM Nightmare: When Harbor Runs Out of Disk Space
At exactly 2:15 AM, my phone started ringing off the hook. The on-call team was frantically pinging me with blood-red screenshots from the CI/CD pipeline: Error: write /var/lib/docker/tmp/docker-builder...: no space left on device. Cascading right after, nearly 40 deployments across the production Kubernetes cluster were stuck in ImagePullBackOff.
I rushed to open the Harbor admin dashboard. The page spun endlessly. The 2TB disk storing artifacts had hit 99.8% capacity. Worse yet, the replication process to the DR backup cluster had quietly hung for six hours without triggering a single alert on any channel.
Running a registry for over 60 engineers without metrics is like driving blind at night. Waiting for a major outage before digging through raw text logs is already too late.
How Harbor Exposes Metrics
Starting with version 2.2, Harbor includes a built-in native Prometheus endpoint. You do not need to install an external exporter.
The system categorizes metrics into two primary groups:
- Harbor Core Metrics: Reports project quota status, actual storage usage, artifact counts, and replication task progress.
- Registry Metrics: Tracks HTTP status codes (200, 404, 500), latency, and the frequency of
docker pushanddocker pullcommands.
Harbor aggregates all this information onto a single port (defaulting to 9090) at the /metrics path. Prometheus periodically scrapes this data into its TSDB, while Grafana queries it to render dashboards.
Step-by-Step Integration with Prometheus & Grafana
Step 1: Enable the Metrics Endpoint in harbor.yml
Access the server hosting Harbor and navigate to the configuration directory:
cd /opt/harbor
sudo vim harbor.yml
Locate the metric block and update it as shown below:
metric:
enabled: true
port: 9090
path: /metrics
Save the file and apply the updated container configuration:
sudo ./prepare
sudo docker compose down
sudo docker compose up -d
Run a test curl command on localhost to verify the exporter:
curl -s http://127.0.0.1:9090/metrics | head -n 20
If the terminal returns a list prefixed with # HELP harbor_..., the endpoint is up and running.
Step 2: Define the Scrape Target in Prometheus
Open the prometheus.yml file on your monitoring server and add a new job:
scrape_configs:
- job_name: 'harbor-monitoring'
scrape_interval: 15s
scrape_timeout: 10s
static_configs:
- targets: ['192.168.10.20:9090']
labels:
env: 'production'
service: 'registry-harbor'
Reload Prometheus without interrupting the service:
curl -X POST http://localhost:9090/-/reload
Navigate to Status > Targets in the Prometheus web UI. Once the harbor-monitoring target turns green with an UP state, you are all set.
Step 3: Set Up Essential PromQL Queries
Here are 3 must-have queries to include on your Grafana Dashboard.
1. Storage Quota Usage by Project
Prevent scenarios where a single repository suddenly bloats and crashes the entire cluster. Query the percentage of storage used:
# Percentage of storage used per project
(harbor_project_quota_used_bytes / harbor_project_quota_total_bytes) * 100
If projects do not have fixed quotas assigned, track the actual storage consumption in gigabytes (GB) instead:
# Storage consumption converted to GB
harbor_project_quota_used_bytes{project_name="core-banking"} / 1024 / 1024 / 1024
2. Image Pull and Push Frequency
This metric provides early detection of CI/CD crash loops or a worker node trapped in an infinite pull loop:
# Image pull rate (req/s)
sum(rate(harbor_registry_http_requests_total{handler="/v2/{name:.*}/manifests/{reference:.*}", method="GET"}[5m]))
# New image push rate (req/s)
sum(rate(harbor_registry_http_requests_total{handler="/v2/{name:.*}/blobs/uploads/{uuid:.*}", method="PUT"}[5m]))
3. Cross-Cluster Replication Status
When synchronizing to a disaster recovery (DR) cluster, any failed task must be caught immediately:
# Number of failed replication tasks in the last 10 minutes
increase(harbor_replication_task_total{status="failed"}[10m])
Step 4: Configure Alert Rules and Filter Out Noise
Alert fatigue quickly dulls an on-call team’s reflexes. Initially, I set an 80% storage threshold with a hold duration of for: 1m. As a result, every time developers ran multi-stage builds or cleared caches, Telegram went off in the middle of the night. The team grew exhausted and gradually muted notifications.
The solution is to split the alert into two tiers with sensible hold durations:
groups:
- name: harbor_alerts
rules:
- alert: HarborStorageQuotaWarning
expr: (harbor_project_quota_used_bytes / harbor_project_quota_total_bytes) * 100 > 85
for: 30m
labels:
severity: warning
annotations:
summary: "Harbor project {{ $labels.project_name }} has reached 85% capacity"
description: "Project storage has exceeded 85% for 30 consecutive minutes. Plan an artifact cleanup."
- alert: HarborStorageQuotaCritical
expr: (harbor_project_quota_used_bytes / harbor_project_quota_total_bytes) * 100 > 95
for: 5m
labels:
severity: critical
annotations:
summary: "Harbor project {{ $labels.project_name }} is almost out of disk space"
description: "Storage has exceeded 95%. Purge old images or increase the quota immediately."
- alert: HarborReplicationFailed
expr: increase(harbor_replication_task_total{status="failed"}[5m]) > 0
for: 2m
labels:
severity: critical
annotations:
summary: "Replication task failed"
description: "Failed to replicate images to the DR cluster. Check network connectivity or authentication tokens."
Holding the 85% threshold for 30 minutes eliminates transient spikes caused by temporary builds. Conversely, triggering at 95% for 5 minutes immediately wakes up the on-call engineer when a cluster outage is imminent.
Conclusion
Don’t wait for the Docker daemon to throw no space left on device before scrambling to clean up disks. Setting up metrics for storage quotas, request rates, and replication status takes only about 20 minutes of configuration. In return, you gain complete visibility into storage health and keep your CI/CD pipelines running smoothly.

