Posted inMonitoring

Docker Swarm Monitoring: ‘Diagnosing’ Replicas and Rolling Updates with Prometheus & cAdvisor

Monitoring tutorial - IT technology blog
Monitoring tutorial - IT technology blog

The Nightmare of Running Docker Swarm Without Data

When I first deployed Docker Swarm for a project, I stayed up all night because of a silly mistake. At that time, I thought Swarm’s orchestration was so good that if a service died, it would resurrect itself, so there was no need to worry. Reality was much harsher. A service hit a CrashLoopBackOff error due to a misconfiguration, causing Swarm to constantly restart and crash. Server CPU spiked to 95%, logs flooded the disk, and I had no idea until a customer called to complain.

Managing a cluster of 10-15 nodes without monitoring is like driving in the fog. I used to have to SSH into each Manager Node and manually type docker service ls just to check the status. This manual approach was incredibly time-consuming and error-prone. Now, with just a glance at the dashboard, I immediately know which service is missing replicas or which node is overloaded.

The Trio: Prometheus, cAdvisor, and Docker Swarm

To monitor effectively, we need an automated data collection mechanism instead of isolated installations. I chose to leverage the power of Swarm itself to deploy the monitoring stack.

  • cAdvisor (Container Advisor): This is the “undercover agent” on each node. It looks into every container to extract CPU, RAM, and Network metrics. In Swarm, I run cAdvisor in mode: global to ensure no node is missed.
  • Prometheus: Acts as the central brain. It periodically polls cAdvisor to pull data (pull mechanism) and stores it in a time-series database.
  • Docker Engine Metrics: Since version 17.05, Docker can export Prometheus-standard metrics. Enabling this feature helps you track the actual state of the entire cluster.

Configuring Docker Engine to Export Data

By default, Docker locks the metrics port. To let Prometheus read replica counts, you need to edit the /etc/docker/daemon.json file on all nodes.

{
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

After editing, restart Docker with the following command:

sudo systemctl restart docker

An important note: Opening port 0.0.0.0:9323 can pose a security risk if the server has a public IP. You should use a firewall (UFW/Iptables) to allow only internal IPs within the cluster to access this port.

Deploying the Monitoring Stack

Instead of running individual commands, I group everything into a monitoring-stack.yml file. The beauty of this approach is its high consistency. Here is a simplified configuration for you to visualize:

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - "9090:9090"
    networks:
      - monitor-net
    deploy:
      placement:
        constraints: [node.role == manager]

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    networks:
      - monitor-net
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    deploy:
      mode: global
      resources:
        limits:
          memory: 128M

networks:
  monitor-net:
    driver: overlay

The Power of Service Discovery in Prometheus

In a Swarm environment, containers frequently hop between nodes. You cannot use static IPs in the configuration file because they change constantly. Prometheus solves this with the dockerswarm_sd_configs feature.

Here is how I configure Prometheus to automatically discover nodes:

scrape_configs:
  - job_name: 'docker-swarm'
    dockerswarm_sd_configs:
      - host: unix:///var/run/docker.sock
        role: nodes
    relabel_configs:
      - target_label: __address__
        replacement: 127.0.0.1:9323

Don’t forget to mount the docker.sock file into the Prometheus container. Without this step, Prometheus will be “locked out” and unable to query Docker Swarm for the list of active nodes.

Monitoring Rolling Updates and Replicas in Practice

Once the data is flowing in, I usually focus on these three practical scenarios:

1. Alerting on Missing Replicas

If you request 5 replicas but only 3 are running, the system is at risk. The following query helps you identify services with missing replicas:

engine_daemon_swarm_service_tasks_total{state="running"}

2. Controlling Rolling Updates

When running docker service update, sometimes a new version has bugs that cause the update process to hang. I monitor the engine_daemon_container_states_containers metric. If I see the restarting state spike after an update, I perform an immediate rollback to avoid service disruption.

3. Checking Node Health

The dockerswarm_node_status metric is extremely useful. Once, one of my nodes lost network connectivity, but Swarm continued to route traffic to it, causing constant 502 errors. Thanks to Prometheus alerts, I detected and resolved the issue in less than 2 minutes.

Conclusion

Setting up monitoring isn’t just about pretty charts. It’s a tool that helps you sleep better at night. Instead of living in anxiety, let Prometheus guard your system. If your cluster is running in production without cAdvisor and Prometheus, take 30 minutes to set them up today. It will save you hours of exhausting debugging later.

Share:
Tags: