Posted inMonitoring

Node Exporter: Collecting Linux System Metrics for Prometheus

Monitoring tutorial - IT technology blog
Monitoring tutorial - IT technology blog

What Is Node Exporter and Why Do You Need It?

Prometheus is great at collecting and storing time-series metrics, but it can’t read Linux hardware or OS information on its own. That’s where Node Exporter comes in — it runs on each server, reads system data, and exposes an HTTP endpoint for Prometheus to scrape periodically.

Simply put: Node Exporter is an “agent” that sits on each machine, reads Linux’s /proc and /sys filesystems, and translates that data into a format Prometheus understands.

My monitoring setup runs Prometheus + Grafana across 15 servers, and this stack has helped catch incidents before users even notice. For example, a staging server’s disk hit 94% at 2 AM — Alertmanager fired a Telegram alert immediately, and we handled it before the workday started. It all begins with a proper Node Exporter installation.

This article focuses specifically on Node Exporter: how to install it, which collectors are worth enabling, and how to verify it’s running correctly. Connecting Prometheus and Grafana is covered in a separate post.

Installing Node Exporter

Method 1: Manual Installation from Binary (Recommended)

This approach gives you full version control and works on any distro. Visit the Node Exporter releases page to grab the latest version link:

# Download Node Exporter (replace version as needed)
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz

# Extract the archive
tar xvf node_exporter-1.8.2.linux-amd64.tar.gz

# Copy the binary to /usr/local/bin
sudo cp node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/
sudo chmod +x /usr/local/bin/node_exporter

Create a Dedicated Service User

Running Node Exporter as root is a bad idea. Create a system user with no shell and no home directory:

sudo useradd --no-create-home --shell /bin/false node_exporter

Create a systemd Service

With a service unit, Node Exporter will start automatically after reboot and is easy to manage with systemctl:

sudo nano /etc/systemd/system/node_exporter.service

File contents:

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --collector.systemd \
  --collector.processes
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
# Reload and enable the service
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

# Check service status
sudo systemctl status node_exporter

Method 2: Install via Package Manager (Ubuntu/Debian)

Some distros ship a package for Node Exporter, but the version is often significantly behind upstream:

# Ubuntu 22.04+
sudo apt install prometheus-node-exporter -y

# Service is created and started automatically
sudo systemctl status prometheus-node-exporter

The downside is that the binary lives at /usr/bin/prometheus-node-exporter instead of /usr/local/bin/node_exporter, and the service name differs too. When your team works across multiple distros, this inconsistency makes debugging confusing. I generally prefer the manual install to keep all servers uniform.

Configuring Collectors

Node Exporter ships with over 50 collectors, and the most important ones are enabled by default. A few useful collectors need to be activated manually.

Default Collectors (Always Enabled)

  • cpu — CPU usage, iowait, steal time
  • meminfo — RAM total, used, buffers, cached
  • diskstats — Disk I/O (read/write ops, throughput)
  • filesystem — Disk space usage per mount point
  • netdev — Network interface statistics
  • loadavg — 1/5/15-minute load averages
  • stat — Kernel statistics (context switches, interrupts)

Additional Collectors to Enable

Add these to the ExecStart line in your systemd unit file:

ExecStart=/usr/local/bin/node_exporter \
  --collector.systemd \
  --collector.processes \
  --collector.interrupts \
  --collector.tcpstat \
  --collector.cpu.info

What each option does:

  • --collector.systemd — Tracks systemd service states (active/failed). Without this, you can’t alert when a service crashes.
  • --collector.processes — Number of running processes and zombie processes
  • --collector.tcpstat — TCP connection states (ESTABLISHED, TIME_WAIT, etc.)
  • --collector.cpu.info — CPU model and frequency information

Disabling Unnecessary Collectors

To reduce the volume of metrics Prometheus has to process, disable collectors you don’t use:

ExecStart=/usr/local/bin/node_exporter \
  --no-collector.infiniband \
  --no-collector.ipvs \
  --collector.systemd

Configuring Network Binding

By default, Node Exporter listens on all interfaces on port 9100. In production, bind it to the internal interface only:

ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address="10.0.0.5:9100" \
  --collector.systemd

Replace 10.0.0.5 with the server’s internal IP, then reload:

sudo systemctl daemon-reload
sudo systemctl restart node_exporter

Verification and Monitoring

Confirm Node Exporter Is Running

# Check which port is listening
ss -tlnp | grep 9100

# Or use curl to view metrics directly
curl http://localhost:9100/metrics | head -50

Output like this means everything is working:

# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 12345.67
node_cpu_seconds_total{cpu="0",mode="iowait"} 23.45

Check Key Metrics

# View CPU metrics
curl -s http://localhost:9100/metrics | grep "^node_cpu"

# View disk usage
curl -s http://localhost:9100/metrics | grep "node_filesystem_avail_bytes"

# View available RAM
curl -s http://localhost:9100/metrics | grep "node_memory_MemAvailable_bytes"

Open Firewall if Needed

The Prometheus server needs to reach port 9100 on each node:

# UFW (Ubuntu)
sudo ufw allow from <prometheus-server-ip> to any port 9100

# Firewalld (CentOS/RHEL)
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="<prometheus-server-ip>" port protocol="tcp" port="9100" accept'
sudo firewall-cmd --reload

Add to Prometheus Config

On the Prometheus server, edit prometheus.yml:

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets:
          - '10.0.0.5:9100'   # server 1
          - '10.0.0.6:9100'   # server 2
          - '10.0.0.7:9100'   # server 3
    # Add labels for easier filtering in Grafana
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

Reload Prometheus:

curl -X POST http://localhost:9090/-/reload

Verify in the Prometheus UI

Open your browser to http://<prometheus-server>:9090/targets — nodes showing UP in green are connected successfully. If you see DOWN, check your firewall rules and IP addresses.

Run a few PromQL queries to confirm data is coming in:

# Total CPU usage (excluding idle)
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Available RAM (GB)
node_memory_MemAvailable_bytes / 1024 / 1024 / 1024

# Remaining disk space (%)
(node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100

Common Troubleshooting

A few issues I’ve run into when setting up Node Exporter across multiple servers:

  • Service won’t start: Run journalctl -u node_exporter -n 20 to check the logs — usually caused by an incorrect binary path or missing file permissions.
  • Port already in use: Use ss -tlnp | grep 9100 to see which process is holding the port.
  • Prometheus can’t scrape: Try curl http://<node-ip>:9100/metrics from the Prometheus server. If it fails to connect, it’s a network/firewall issue, not a Node Exporter problem.
  • No systemd metrics: The node_exporter user needs read access to D-Bus — some distros require adding the user to the systemd-journal group.

Share:
Tags: