The Problem: The Nightmare of SSHing into Every Server to Troubleshoot
Your infrastructure is running on 10 servers: 3 Nginx web nodes, 2 backend API clusters, a PostgreSQL database, and 2 gateways. At 3 PM, users start complaining about persistent 502 errors during checkout. You find yourself opening a dozen terminal tabs simultaneously, typing tail -f /var/log/nginx/error.log on each machine to grep for request IDs. This process is both chaotic and prone to missing critical issues.
When your infrastructure scales to 30 or 50 servers, this manual approach completely falls apart. Without centralized logging, your Mean Time to Resolution (MTTR) can stretch from minutes into hours. You need to aggregate all log data into a single location and search it instantly with a single query.
Graylog Cluster Architecture
Graylog acts as a centralized data ingestion hub. The system collects logs from every corner of your infrastructure, parses structured fields, and stores them in an indexed data warehouse.
A complete Graylog cluster consists of three main components:
- MongoDB: Stores system configurations, user information, alert streams, and dashboard metadata. It does not store actual log contents.
- OpenSearch: The core engine for log storage and full-text search queries. The speed of filtering through millions of log lines directly depends on the amount of RAM allocated to OpenSearch.
- Graylog Server: The central processing engine. This service receives logs from clients via Syslog/GELF/Beats, parses the incoming data, forwards it to OpenSearch, and serves the Web UI.
Step-by-Step Guide to Deploying Graylog on CentOS Stream 9
Step 1: Prepare the Environment and Install Java OpenJDK
Both Graylog and OpenSearch run on the JVM. On CentOS Stream 9, you can directly use OpenJDK 17 available in the default repository:
sudo dnf update -y
sudo dnf install -y java-17-openjdk-headless pwgen wget curl lsofStep 2: Install MongoDB 6.0
Graylog 6.x requires MongoDB 6.0 or newer. Add the MongoDB repository configuration file:
cat <<EOF | sudo tee /etc/yum.repos.d/mongodb-org.repo
[mongodb-org-6.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/9/mongodb-org/6.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-6.0.asc
EOF
sudo dnf install -y mongodb-org
sudo systemctl daemon-reload
sudo systemctl enable --now mongodCheck the MongoDB service status using systemctl status mongod to ensure it is in the active (running) state.
Step 3: Install and Configure OpenSearch 2.x
Add the official OpenSearch repository:
cat <<EOF | sudo tee /etc/yum.repos.d/opensearch.repo
[opensearch-2.x]
name=OpenSearch 2.x Repository
baseurl=https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/yum
gpgcheck=1
gpgkey=https://artifacts.opensearch.org/publickeys/opensearch.pgp
enabled=1
EOF
sudo dnf install -y opensearchOpen /etc/opensearch/opensearch.yml and configure it for a single-node setup:
cluster.name: graylog-cluster
node.name: node-1
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 127.0.0.1
http.port: 9200
discovery.type: single-node
action.auto_create_index: false
plugins.security.disabled: trueFor a server with 8GB of RAM, adjust the JVM heap size in /etc/opensearch/jvm.options to -Xms4g and -Xmx4g (roughly 50% of system RAM) to optimize server performance. Then start the service:
sudo systemctl daemon-reload
sudo systemctl enable --now opensearchStep 4: Install and Configure Graylog Server
Install the Graylog 6.0 repository and package:
sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-6.0-repository_latest.rpm
sudo dnf install -y graylog-serverGraylog requires two secret keys to start up:
password_secret: Used to encrypt user session cookies.root_password_sha2: The SHA-256 hash of theadminaccount password.
Quickly generate these two values via the terminal:
# Generate a random 96-character secret key
pwgen -N 1 -s 96
# Hash the admin password (replace AdminPassword@2026 with your own password)
echo -n "AdminPassword@2026" | sha256sum | cut -d" " -f1Open /etc/graylog/server/server.conf and paste the corresponding values:
password_secret = <generated_pwgen_secret>
root_password_sha2 = <generated_sha256_hash>
http_bind_address = 0.0.0.0:9000Save the file and enable Graylog:
sudo systemctl daemon-reload
sudo systemctl enable --now graylog-serverStep 5: Configure the Firewall and Verify the Web UI
Open port 9000 for the Web interface and port 1514 UDP for receiving logs from client machines using Firewalld:
sudo firewall-cmd --add-port=9000/tcp --permanent
sudo firewall-cmd --add-port=1514/udp --permanent
sudo firewall-cmd --reloadOpen your browser and navigate to http://<SERVER_IP>:9000. Log in using the username admin and the plaintext password you hashed in Step 4.
Step 6: Create an Input and Configure Clients to Forward Logs
Configure a log ingestion input via the Graylog Web UI:
- Navigate to System > Inputs.
- Select Syslog UDP from the dropdown menu and click Launch new input.
- Enter Title:
Linux Syslog Input, Port:1514, Bind address:0.0.0.0, then click Save.
Switch to the client servers you want to monitor. Add an rsyslog configuration to forward logs to the Graylog server:
echo "*.* @<GRAYLOG_SERVER_IP>:1514;RSYSLOG_SyslogProtocol23Format" | sudo tee /etc/rsyslog.d/50-graylog.conf
sudo systemctl restart rsyslogReturn to the Search tab in Graylog, and you will see log messages streaming in from the client in real time.
Conclusion
A centralized logging system frees you from the tedious routine of hunting for errors across individual terminal sessions. Alongside metrics visualized in Grafana, Graylog and OpenSearch ensure all logs—from authentication and kernel events to application errors—are neatly consolidated, ready for rapid troubleshooting and automated alerting via Telegram or Slack.
