Posted inMonitoring

Monitoring Argo CD Application Sync and Health Status with Prometheus and Grafana

Why Can GitOps Deployments Still Fail Silently?

In a GitOps workflow, Argo CD acts as the mission control center. Every manifest change pushed to Git is automatically pulled and applied to the Kubernetes cluster by Argo CD. However, real-world operations are not always smooth sailing.

Applications can get stuck in an OutOfSync state or degrade into Degraded. Without timely alerting, new releases can stall, or worse, take down production services without anyone noticing.

Our monitoring setup uses Prometheus and Grafana to manage 15 clusters running over 80 microservices. In the past, a developer on our team accidentally messed up the indentation in a Helm chart’s values-prod.yaml file. Git accepted the commit, but Argo CD silently failed to sync for three days straight. It wasn’t until QA tested the new feature that everyone realized the updated code had never actually rolled out.

To eliminate these surprises, the objective is straightforward: collect all Argo CD Sync Status and Health Status metrics into Prometheus, visualize them on a Grafana dashboard, and trigger alerts via Slack or Telegram.

Scraping Metrics from Argo CD with Prometheus Operator

Core Argo CD components such as argocd-server, argocd-repo-server, and argocd-application-controller natively export metrics on port 8082 or 8083. If your cluster already runs kube-prometheus-stack, setting this up takes just two ServiceMonitor files.

First, verify whether the metrics Services are available in the argocd namespace:

kubectl get svc -n argocd -l app.kubernetes.io/name=argocd-metrics

If the output shows a service listening on port 8082, create the argocd-servicemonitor.yaml file so Prometheus Operator can automatically scrape it:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-metrics
  namespace: argocd
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-metrics
  endpoints:
  - port: metrics
    interval: 30s
    scrapeTimeout: 10s

Along with the controller, argocd-server and argocd-repo-server also warrant close monitoring. Metrics from the repo-server will tell you if Git repos are experiencing bottlenecks or hitting rate limits:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-server-metrics
  namespace: argocd
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-server-metrics
  endpoints:
  - port: metrics
    interval: 30s

Apply both configurations to your cluster:

kubectl apply -f argocd-servicemonitor.yaml

Breaking Down Metrics and Crafting PromQL Queries

1. The Four Most Valuable Argo CD Metrics

Argo CD exposes a wealth of data, but for day-to-day operations, you only need to focus on these four metrics:

  • argocd_app_info: The most critical metric. It includes key labels such as sync_status (Synced, OutOfSync) and health_status (Healthy, Degraded, Progressing, Suspended). This metric always returns a value of 1.
  • argocd_app_sync_total: The total count of sync operations broken down by phase (phase="Succeeded", phase="Failed", or phase="Error").
  • argocd_app_reconcile_count: How frequently the controller reconciles differences between the Git manifests and live cluster state.
  • argocd_git_request_total: The number of requests sent to GitHub or GitLab. Very helpful when debugging HTTP 429 Too Many Requests errors.

2. Essential Everyday PromQL Queries

List all applications currently in a Degraded state:

sum(argocd_app_info{health_status="Degraded"}) by (name, dest_server, namespace)

Filter applications with configuration drift between Git and the cluster (OutOfSync):

sum(argocd_app_info{sync_status="OutOfSync"}) by (name, dest_server, namespace)

Calculate the sync failure rate over the last 15 minutes (if this exceeds 5%, your CI/CD pipeline likely has an issue):

sum(rate(argocd_app_sync_total{phase=~"Failed|Error"}[15m])) 
/ 
sum(rate(argocd_app_sync_total[15m])) * 100

3. Configuring PrometheusRules for Alerting

Don’t wait until you open Grafana to discover an app is down. Set up automated alerts so on-call engineers are notified immediately when an issue persists:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: argocd-application-alerts
  namespace: argocd
  labels:
    release: kube-prometheus-stack
spec:
  groups:
  - name: argocd.rules
    rules:
    - alert: ArgoAppDegraded
      expr: argocd_app_info{health_status="Degraded"} == 1
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Application {{ $labels.name }} is Degraded"
        description: "App {{ $labels.name }} on cluster {{ $labels.dest_server }} has been in a degraded state for over 5 minutes."

    - alert: ArgoAppOutOfSync
      expr: argocd_app_info{sync_status="OutOfSync"} == 1
      for: 15m
      labels:
        severity: warning
      annotations:
        summary: "Application {{ $labels.name }} is OutOfSync"
        description: "Cluster and Git state for app {{ $labels.name }} have drifted for over 15 minutes without reconciling."

4. Importing Dashboards into Grafana

Instead of spending hours building charts from scratch, leverage community-proven dashboards built for Argo CD. Navigate to the Grafana UI, go to Dashboards > Import, and enter ID 14584 (Argo CD Official Dashboard) or 13038.

These dashboards provide essential panels out of the box: total application count, Healthy/Sync ratio charts, lists of failing apps, and repo-server latency. Once you select the appropriate Prometheus data source, all the metrics will populate immediately.

Testing in a Real-World Scenario

A monitoring setup isn’t truly complete until you validate it with a simulated failure.

First, verify that Prometheus has successfully discovered the Argo CD scrape targets:

kubectl port-forward svc/kube-prometheus-stack-prometheus -n monitoring 9090:9090

Open http://localhost:9090/targets in your browser. Confirm that the serviceMonitor/argocd/argocd-metrics/0 endpoint shows green and reports UP.

Next, trigger a test incident on a staging app. Update the Git manifest by changing the image tag to a non-existent version, such as auth-service:v999-notfound, and push the change. Argo CD will pull the new commit. The pod will immediately fail with ImagePullBackOff, causing Argo CD to transition the health status to Degraded.

Check Grafana: the Healthy Apps count will drop by one, and the Degraded panel will display auth-service. After 5 minutes, Alertmanager will fire the ArgoAppDegraded alert.

Finally, revert the commit in Git back to the valid image tag. Argo CD will reconcile successfully, the pod will return to a healthy green state, and the alert will automatically resolve. Your team’s GitOps pipeline is now backed by a reliable, 24/7 safety net.

Share:
Tags: