Installing Grafana on CentOS Stream 9: Professional Server Monitoring with Prometheus – ITFROMZERO

Table of Contents

The “Midnight Call” from the Boss When the Server Crashes

Back when I first started, there was a time the company’s core application server crashed right on New Year’s Eve. At that moment, I could only frantically SSH in and type top or df -h in vain. Unfortunately, once a server is down, you can’t get historical data to know when it became overloaded. Was it a memory leak or CPU usage by a strange process? No one knew.

Hard-earned experience shows: Managing systems without monitoring charts is like driving at night without headlights. You only know you’ve hit a wall when the accident has already happened.

Why Manual Log Reading is a “Time Trap”?

Many IT professionals still maintain the habit of digging through /var/log/messages or journalctl only when an incident occurs. This method isn’t wrong, but it’s extremely exhausting for the following reasons:

Loss of predictability: You can’t see RAM consumption trends. For example, a steady 1% increase per hour is a sign of a memory leak that log files can’t easily pinpoint immediately.
Resource waste: Using grep to filter through millions of log lines in 5-10GB files is a nightmare.
Hard to convince management: Bosses won’t understand dry log lines. They need visual charts to approve budgets for server upgrades.

The Grafana – Prometheus Duo: A Top Choice for DevOps

I used to use Zabbix, but it’s quite heavy and the interface feels a bit “ancient.” When I switched to working with Docker and CentOS Stream 9, I prioritized the Prometheus and Grafana duo.

Prometheus acts as a powerful Time Series Database. Meanwhile, Grafana is the “beauty queen” of visualization, turning lifeless numbers into stunning dashboards. In practice, I deployed this duo to monitor a cluster of 20 Linux servers, reducing troubleshooting time from 30 minutes to just 2 minutes.

Practical Implementation Steps on CentOS Stream 9

We will follow this roadmap: Install Node Exporter (data collection) -> Install Prometheus (storage) -> Install Grafana (visualization) -> Security configuration.

Step 1: Installing Node Exporter and Prometheus

Node Exporter is a lightweight tool running on the server to push CPU, RAM, and Disk metrics. First, create a dedicated user to ensure system security:

# Create service user
sudo useradd --no-create-home --shell /bin/false node_exporter

# Download and install Node Exporter v1.7.0
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar -xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

Create a service file to manage Node Exporter more easily:

sudo nano /etc/systemd/system/node_exporter.service
# [File content: User=node_exporter, ExecStart=/usr/local/bin/node_exporter]
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Step 2: Installing Grafana from the Official Repository

Grafana supports RPM repos very well on CentOS 9, making future dnf update tasks effortless. Create the repo file:

sudo nano /etc/yum.repos.d/grafana.repo

Add the official Grafana Labs repository configuration, then run the installation command:

sudo dnf install grafana -y
sudo systemctl enable --now grafana-server

Step 3: Security Hardening with Firewalld and SELinux

This is a step many juniors often forget, leading to inaccessible dashboards. CentOS Stream 9 blocks most non-standard ports by default.

Firewalld Configuration: Open port 3000 for Grafana and 9090 for Prometheus.

sudo firewall-cmd --permanent --add-port={3000,9090}/tcp
sudo firewall-cmd --reload

Handling SELinux: Don’t rush to disable SELinux. If Grafana needs to connect to a database on another server, grant specific permissions:

sudo setsebool -P httpd_can_network_connect 1

Note: If you see a “Permission Denied” error in the logs despite having set file permissions, 99% of the time it’s SELinux blocking access to the data directory.

Step 4: Connecting and Creating a Dashboard in 5 Minutes

Access http://server-IP:3000 with the credentials admin/admin. After changing the password, follow these steps:

In the Data Sources menu, select Prometheus.
Enter the URL: http://localhost:9090 and click Save & Test.
Pro tip: Don’t build charts from scratch. Go to the Import section and enter ID 1860. This is the highly detailed “Node Exporter Full” template trusted by the community.

Real-world Operational Experience

After years of managing systems on the RHEL family, I’ve drawn three important lessons:

Never disable SELinux: Use audit2allow to debug instead of setenforce 0. Disabling SELinux means removing your server’s last layer of armor.
Control Retention: Prometheus keeps data for 15 days by default. If your server only has 20GB of disk space, reduce it to 7 days using the parameter --storage.tsdb.retention.time=7d to prevent crashes due to a full disk.
Prioritize Alerting: Pretty dashboards are just for show. Configure alerts via Telegram or Slack. A “CPU > 90% for 5 minutes” notification sent to your phone will save you faster than any chart.

Mastering Grafana on CentOS Stream 9 not only makes your job easier but also elevates your professionalism in the eyes of your colleagues. Good luck with your deployment!