Centralized Logging with Rsyslog on CentOS Stream 9: Streamline logs for peace of mind

The nightmare of “searching for a needle in a haystack” across dozens of log files

As a system operator, you’ve likely experienced the feeling of a perfectly running system suddenly “dropping dead.” The most frustrating part is not knowing why. In those moments, the first step is always digging through logs to find the root cause.

If you’re only managing 1 or 2 servers, SSHing into each and running tail -f /var/log/messages takes just a few minutes. But imagine a system with 20, 50, or hundreds of VMs running microservices. This manual process becomes a nightmare. You’ll spend hours just switching between terminal windows.

I remember a web server cluster hitting 502 errors constantly at 2 AM. I had to open 5 terminal tabs, fighting off sleep while checking each one for upstream errors. It was incredibly frustrating. Even worse, if a server suffers sudden hardware failure, all the evidence (logs) vanishes along with the machine. That’s when I realized: Centralized Logging isn’t an option; it’s a necessity.

Why are logs so scattered and difficult to manage?

By default, Linux distributions like CentOS Stream 9 store logs locally. Every service—from SSH and Mail to the Kernel—writes to separate files in /var/log/. This approach has three critical weaknesses:

• Fragmented data: To understand the “big picture” of the system, you must painstakingly piece together information from 5 to 10 different sources.
• Security risks: Once hackers gain root access, they often wipe log files to cover their tracks. If logs only exist locally, you lose any chance of investigation.
• System crash risk: Excessive logging can fill up the /var partition, causing other services to halt because there’s no storage space left.

Overview of popular solutions

There are many powerful tools available to solve this problem:

1. ELK Stack (Elasticsearch, Logstash, Kibana): The gold standard for searching and visualization. However, ELK is extremely resource-heavy. A basic ELK cluster needs at least 4-8GB of RAM just to start.
2. Graylog: Easier to manage than ELK but still requires significant infrastructure to run stably.
3. Loki + Grafana: Very lightweight, perfect for Docker/Kubernetes environments, but requires learning the new LogQL query language.

Rsyslog: The “affordable and efficient” built-in choice

For medium-scale systems, or if you prioritize stability and low overhead, Rsyslog is the perfect fit. This tool consumes less than 50MB of RAM yet remains incredibly powerful. It comes pre-installed on most RHEL-based distributions like CentOS Stream 9, AlmaLinux, and Rocky Linux.

When CentOS 8 reached EOL, I had to migrate a server farm to CentOS Stream 9 quickly. The first thing I did was rebuild the logging cluster using Rsyslog. It has run reliably for years with almost zero maintenance.

Step 1: Configure the Log Server (Receiver)

We need to designate one server as the central hub. All configurations are located in the /etc/rsyslog.conf file.

Open the file with your preferred editor:

sudo vi /etc/rsyslog.conf

Find and uncomment the following lines to let Rsyslog listen on port 514 for both UDP and TCP:

# For UDP
module(load="imudp")
input(type="imudp" port="514")

# For TCP
module(load="imtcp")
input(type="imtcp" port="514")

To prevent client logs from getting mixed with server logs, I use a template for automatic categorization. Add this code to the end of the file:

$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& ~

This configuration automatically sorts logs into directories based on the hostname and application name. Save the file and restart the service:

sudo systemctl restart rsyslog
sudo systemctl enable rsyslog

Step 2: Open Firewall on CentOS Stream 9

Don’t forget to open port 514 on the firewall, otherwise, clients will be blocked from sending logs.

sudo firewall-cmd --permanent --add-port=514/udp
sudo firewall-cmd --permanent --add-port=514/tcp
sudo firewall-cmd --reload

Step 3: Configure the Log Client (Sender)

Now, switch to the machines you want to monitor. Open the /etc/rsyslog.conf file there as well:

sudo vi /etc/rsyslog.conf

Add the following lines to the very end (remember to replace the IP with your Log Server address):

# Send logs via UDP (Fast but may lose packets)
*.* @192.168.1.10:514

# Send logs via TCP (Slower but more reliable)
*.* @@192.168.1.10:514

Restart Rsyslog on the client to apply the changes:

sudo systemctl restart rsyslog

Step 4: Verify the results

On the Log Server, check the /var/log/remote/ directory:

ls -R /var/log/remote/

To test immediately, you can send a message from the Client machine:

logger "Testing centralized logging for itfromzero!"

Return to the Log Server, and you’ll see this line appear instantly in that client’s log file. There’s a great sense of satisfaction seeing logs from everywhere pouring into one place—it feels like you have total control over your entire system.

A few “hard-earned” tips from real-world experience

When deploying to a production system, you should pay attention to three things:

1. Disk space management: Centralized logs grow very quickly. Set up logrotate immediately to compress and delete old logs periodically (e.g., keep logs for 30 days).
2. Transmission security: By default, Rsyslog sends data in plain text. If sending over the Internet, you must configure TLS/SSL to prevent eavesdropping on sensitive information.
3. SELinux: CentOS Stream 9 is very strict with SELinux. If you use a non-default port (other than 514), don’t forget to update the port label using the semanage port command.

Setting up centralized logging isn’t difficult, but the benefits are invaluable. It helps you sleep better knowing that if an issue occurs, all the evidence is neatly gathered in one place, waiting for you to handle it.