A Real-World Scenario: A 2 AM Outage and the Challenge of Tracing Covert Changes
At 2:15 AM, PagerDuty started alerting non-stop. The Nginx reverse proxy cluster was completely down, and over 15,000 incoming HTTPS requests per minute were failing with 502 Bad Gateway errors. After SSHing into the production CentOS Stream 9 server, I discovered that the certificate file /etc/pki/tls/certs/api.crt had its permissions changed to 000. Running history showed a completely empty terminal—the culprit had either prepended a space to their commands or executed unset HISTFILE before exiting the shell.
Checking journalctl -u nginx and /var/log/messages yielded only a curt message: cannot load certificate: Permission denied. There was absolutely no trace of who had run the chmod command on the certificate file in the middle of the night.
That is the price of neglecting the Linux Audit Framework when provisioning new servers. Syslog and journald only passively record what applications report. In contrast, auditd hooks directly into the Linux kernel via the Audit Subsystem. Whenever a process makes a system call (reading, overwriting, deleting files, or changing ownership), the kernel immediately logs it. Most importantly, the auid (Audit UID) field allows the kernel to trace precisely which user account initiated the command, even if they ran sudo su - to switch to root.
Installing Auditd on CentOS Stream 9
On a CentOS Stream 9 Minimal installation, the audit package is usually present, but utility tools may be missing. Install the complete package suite with:
sudo dnf install -y audit audit-libs
Enable and start the daemon:
sudo systemctl enable auditd
sudo systemctl start auditd
Important note: Systemd on RHEL 9 and CentOS Stream 9 blocks the systemctl restart auditd command to prevent interruptions to the kernel’s security logging process. To restart the service, you must use:
sudo service auditd restart
Check the status of the audit subsystem in the kernel:
sudo auditctl -s
If the output returns enabled 1 and a valid pid, the kernel is ready to capture audit events.
Detailed Configuration: Battle-Tested Rules for Production Servers
The Auditd architecture consists of two main parts: /etc/audit/auditd.conf (manages log files, storage limits, and rotation behavior) and the /etc/audit/rules.d/ directory (contains monitoring rules).
1. Preventing Disk Overflow in auditd.conf
On high-traffic server clusters, audit logs can consume tens of gigabytes per day. If the /var/log partition fills up, services on the server risk crashing or freezing. Open the configuration file:
sudo vi /etc/audit/auditd.conf
Configure safe storage thresholds:
# Maximum size for each log file (MB)
max_log_file = 50
# Keep a maximum of 10 rotated log files (500MB total max)
num_logs = 10
max_log_file_action = ROTATE
# Warn via syslog when remaining disk space falls below 100MB
space_left = 100
sace_left_action = SYSLOG
# Suspend audit logging when remaining disk space falls below 50MB to protect the server
admin_space_left = 50
admin_space_left_action = SUSPEND
disk_full_action = SUSPEND
This configuration caps total audit log usage at 500MB while triggering early syslog warnings before disk space is exhausted.
2. Creating a Persistent Monitoring Ruleset
Avoid defining rules directly using the auditctl command, as they will be lost after a reboot. Instead, declare them persistently in /etc/audit/rules.d/audit.rules:
sudo vi /etc/audit/rules.d/audit.rules
Paste the following standardized ruleset:
## Flush all existing rules to avoid conflicts
-D
## Increase buffer to 8192 messages (prevents dropped logs during I/O spikes)
-b 8192
## 1. Monitor user accounts and system permissions
-w /etc/passwd -p wa -k identity_changes
-w /etc/shadow -p wa -k identity_changes
-w /etc/group -p wa -k identity_changes
-w /etc/sudoers -p wa -k sudoers_tamper
-w /etc/sudoers.d/ -p wa -k sudoers_tamper
## 2. Monitor Nginx configs and SSL certs
-w /etc/nginx/ -p wa -k nginx_config_tamper
-w /etc/pki/tls/ -p wa -k ssl_certs_tamper
## 3. Capture file permission and ownership syscalls (chmod, chown)
-a always,exit -F arch=b64 -S chmod,fchmod,fchmodat -F auid>=1000 -F auid!=unset -k file_permission_change
-a always,exit -F arch=b64 -S chown,fchown,lchown,fchownat -F auid>=1000 -F auid!=unset -k file_owner_change
## 4. Capture file and directory deletion syscalls
-a always,exit -F arch=b64 -S unlink,unlinkat,rmdir -F auid>=1000 -F auid!=unset -k file_deletion
## 5. Make the configuration immutable (uncomment only when the system is stable)
#-e 2
Key parameters to remember:
-w <path>: Defines a watchpoint on a specific file or directory.-p wa: Filters two permission operations:write(write/modify) andattribute(changes to permissions, ownership, or timestamps).-k <key>: Attaches a custom key/tag to quickly filter events during queries.-F auid>=1000 -F auid!=unset: Filters for real logged-in users (UID 1000 and above), ignoring system daemons (UID < 1000) to prevent log noise.-e 2: Locks the audit configuration in the kernel. Once this flag is set, rules cannot be altered until the server is rebooted.
Load the new rules into the kernel:
sudo augenrules --load
Verify the currently active rules:
sudo auditctl -l
Investigative Hands-on: Tracing Evidence with ausearch and aureport
Avoid using cat or grep directly on /var/log/audit/audit.log. Raw audit logs contain hex-encoded strings that are difficult to parse manually. Instead, leverage the power of ausearch and aureport.
1. Simulating an Unauthorized Action
Log in with a regular user account (e.g., hungnd with UID 1001) and use sudo to modify the Nginx configuration permissions:
sudo chmod 000 /etc/nginx/nginx.conf
2. Tracing the Incident with ausearch
Search for the event using the file_permission_change key, adding the -i (interpret) flag to decode hex values into human-readable text:
sudo ausearch -k file_permission_change -i --recent
The resulting log output reveals the complete context:
type=SYSCALL msg=audit(18/10/2026 02:15:30.120:412) : arch=x86_64 syscall=fchmodat success=yes exit=0 a0=0xffffff9c a1=0x154b200 a2=000 items=1 ppid=2450 pid=2810 auid=hungnd uid=root gid=root euid=root ses=3 comm="chmod" exe="/usr/bin/chmod" key="file_permission_change"
type=PATH msg=audit(18/10/2026 02:15:30.120:412) : item=0 name="/etc/nginx/nginx.conf" inode=135894 dev=fd:00 mode=0100644 ouid=0 ogid=0
What do these key fields tell us?
- comm & exe: The command executed was
/usr/bin/chmod. - name: The affected target file was
/etc/nginx/nginx.conf. - uid=root: The command ran with root privileges.
- auid=hungnd: The originating identity. Even though the user escalated privileges using
sudo, the audit subsystem pinpointed that the user who initiated the session was originallyhungnd.
3. Quick Summaries with aureport
For periodic security audits, aureport produces concise summary reports in seconds.
Generate summary statistics for successful and failed logins:
sudo aureport -l --summary
List operations denied by the kernel (potential signs of permission probing or privilege escalation attempts):
sudo aureport --failed
View a list of files modified within the last 24 hours:
sudo aureport -f -ts yesterday
Auditd cannot prevent malicious actions while they are occurring. However, it delivers what matters most during an incident: incontrovertible forensic evidence to establish accountability and permanently remediate operational vulnerabilities.
