Why nvidia-smi is not enough for GPU management
If you are running AI systems or training Deep Learning models, GPUs are the heart of your infrastructure. An H100 or A100 can cost tens of thousands of dollars, so protecting them is a top priority. Typically, we SSH into the server and run nvidia-smi for a quick check. However, this method has a fatal flaw: it only shows data at a single point in time.
In reality, you can’t stay awake 24/7 to monitor whether a GPU is overheating and thermal throttling at 2 AM. Or if the Power Supply Unit (PSU) is overloaded when model training hits peak power? This is where NVIDIA DCGM Exporter, combined with Prometheus and Grafana, comes into play, allowing you to track the entire operational history automatically.
Quick Start: Activate monitoring in 5 minutes
As long as your server has NVIDIA Drivers and Docker installed, you can push all GPU metrics to a dashboard with a single command. This is the fastest way to check compatibility before official deployment.
docker run -d --gpus all \
--name nvidia-dcgm-exporter \
-p 9400:9400 \
nvcr.io/nvidia/k8s-device-plugin:dcgm-exporter:3.3.5-3.4.0-ubuntu22.04Once the container is running, access http://<Server-IP>:9400/metrics. If the screen displays a series of metrics like DCGM_FI_DEV_GPU_TEMP, your system is ready to stream data.
DCGM Exporter: The brain behind the numbers
NVIDIA Data Center GPU Manager (DCGM) is a specialized suite of tools for managing GPUs in large data centers. DCGM Exporter acts as a high-level “interpreter.” It extracts data from the NVML library and converts it into a format that Prometheus can read.
Why not use community tools? DCGM Exporter is an official product from NVIDIA, providing deep support for metrics that nvidia-smi often overlooks. You can monitor Tensor Core utilization or FP64/FP32 throughput in detail – vital parameters for optimizing model training performance.
Step 1: Install NVIDIA Container Toolkit
About 90% of “container cannot see GPU” errors stem from missing this Toolkit. It allows the Docker engine to communicate directly with the NVIDIA driver at the host level.
# Add official repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
# Install and restart service
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart dockerStep 2: Configure Prometheus for data collection
Now, you need to configure Prometheus to periodically visit port 9400 and “scrape” the data. Add the following configuration to your prometheus.yml file:
scrape_configs:
- job_name: 'nvidia-gpu'
static_configs:
- targets: ['<GPU-SERVER-IP>:9400']
scrape_interval: 15s # This frequency is sufficient for monitoring temperature without stressing the CPUReal-world Experience: Avoiding the Alert Fatigue Trap
My biggest mistake when I first started monitoring was setting alerts that were too sensitive. Receiving 200 Telegram messages a night because the GPU hit 81°C will make you “numb” to real incidents. For models like the A100, a threshold of 80°C under full load is perfectly normal.
- Temperature: Only alert when the GPU exceeds 85°C continuously for 5-10 minutes.
- VRAM: Set a 95% threshold to detect Memory Leak issues in PyTorch code early, before the program crashes (Out of Memory).
- Clock Speed: If the temperature is low but the clock speed drops significantly, check the power supply immediately. The PSU might not be providing enough peak power to the card.
Visualization with Grafana Dashboard
Instead of spending all morning creating your own charts, use Dashboard ID 12239 from Grafana Labs. This is a standard dashboard template pre-optimized by NVIDIA.
This dashboard provides an overview of:
- GPU Utilization: The actual computational percentage.
- Power Usage: Power consumption (helping you calculate operating costs accurately).
- XID Errors: Critical hardware errors from the NVIDIA Driver (crucial for warranty claims).
Pro Tips for Large-Scale System Operators
If you are managing a Kubernetes cluster, don’t install everything manually. Use the NVIDIA GPU Operator via Helm Chart. It automatically manages everything from drivers and container runtimes to exporters across the entire cluster, reducing maintenance effort by 90%.
Additionally, keep an eye out for XID 31 or XID 43 errors. If these error codes appear on Grafana, there’s a high chance your card is experiencing hardware issues or a loose power cable, requiring an immediate physical inspection.
Conclusion
GPU monitoring isn’t just about looking at pretty red and green charts. It’s a tool to proactively protect major assets and maintain stability for AI projects. Spending just 15 minutes on setup today will save you weeks of troubleshooting if hardware issues occur.
