A Monday Morning Nightmare: Nginx Crashes and the Trail Goes Cold
At 9:00 AM on a Monday morning, the system suddenly started throwing nonstop 502 Bad Gateway errors. Customer complaints began pouring in. Checking /etc/nginx/nginx.conf, I discovered that someone had modified an upstream block at 11:15 PM the night before, causing Nginx syntax checks to fail on reload. Digging into /var/log/auth.log revealed that three DevOps engineers had SSHed into the server around that exact time.
When I inspected each user’s ~/.bash_history, the investigation hit a complete dead end. The command history was blank. One person had used a tmux session and disconnected abruptly due to network drops, another shared a deploy account, or the edit command had been executed behind the scenes via a bash script. No one took ownership. Who opened the file? Which process overwrote the data? Who was the real UID behind the action? All traces had vanished into thin air.
Why Ubuntu’s Default Logging Is Easy to Bypass
Many sysadmins rely heavily on syslog, journalctl, or bash history when investigating incidents. Unfortunately, these mechanisms only operate on the surface and are remarkably easy to circumvent:
- Bash history lives in User Space: A user can easily type
set +o history, prefix commands with a space, or simply delete~/.bash_history, erasing their operational footprint without a trace. - Syslog passively waits for applications to report: Nginx or SSH only log predefined application events (like logins or reloads). They are completely blind when someone uses
vim,nano, orsedto peek at a private key inside/etc/ssl/private/. - The trail goes cold after privilege escalation: When someone executes
sudo suto switch to root, their original identity is easily lost. Without a mechanism tracking the Audit User ID (auid) tied to the initial login session, you have no reliable way to identify who is operating behind that root account.
Makeshift Workarounds and Their Real-World Limits
Before turning to Auditd, our team tried several common workarounds, but all came with serious drawbacks:
1. Locking Files with chattr +i
Running chattr +i /etc/nginx/nginx.conf prevents edits and deletions effectively—even root is blocked. However, this approach is only practical for purely static files. Whenever a CI/CD pipeline needs to update the configuration, the deployment fails. Furthermore, it provides zero visibility into who attempted to tamper with the file.
2. File Integrity Monitoring with AIDE or Tripwire
These tools periodically scan file hashes via cron jobs (e.g., every 30 minutes). Their fatal flaw is latency. They only notify you after the file has already been altered, lack real-time alerting, and cannot pinpoint the perpetrator’s PID or UID.
3. Monitoring with inotifywait Scripts
While custom scripts catch file events quickly, they consume excessive CPU and RAM when directories contain many nested files. Under heavy disk I/O bursts, the inotify event queue overflows, causing the script to crash immediately.
The Enterprise Standard: Capturing Kernel-Level Events with Auditd
Instead of playing catch-up at the application layer, the Linux Audit Daemon (Auditd) intercepts events directly inside the Linux Kernel. Regardless of whether a user runs sophisticated scripts or deletes logs in user space, every system call—such as openat, execve, or unlink—is captured intact by the kernel.
Below is a configuration battle-tested on an Ubuntu 22.04 LTS cluster handling approximately 3,500 requests per second. It ensures all sensitive activities are tracked while keeping Auditd’s CPU usage below 1.5%.
Step 1: Install Auditd
Ubuntu Server typically does not have Auditd enabled out of the box. Install the required packages using the following commands:
sudo apt update
sudo apt install -y auditd audispd-plugins
sudo systemctl enable --now auditd
sudo systemctl status auditdStep 2: Configure Audit Rules Following Security Best Practices
Auditd loads rules from /etc/audit/rules.d/. Never edit the temporary /etc/audit/audit.rules file directly, as it gets overwritten on every reboot. Instead, create a dedicated rule file:
sudo nano /etc/audit/rules.d/custom-security.rulesPaste in the following production-ready rules:
## 1. Delete all existing rules from kernel memory
-D
## 2. Increase buffer size to 8192 (prevents log drops during I/O spikes)
-b 8192
## When buffer is full: 1 = send syslog warning, 2 = kernel panic (use 1 for safety)
-f 1
## 3. Monitor changes to user accounts and sudo privileges
-w /etc/passwd -p wa -k identity_changes
-w /etc/shadow -p wa -k identity_changes
-w /etc/group -p wa -k identity_changes
-w /etc/sudoers -p wa -k sudoers_tamper
-w /etc/sudoers.d/ -p wa -k sudoers_tamper
## 4. Monitor Nginx configuration and SSL certificate changes
-w /etc/nginx/ -p wa -k nginx_config_changes
-w /etc/ssl/ -p rwa -k ssl_key_access
## 5. Capture system calls attempting to alter system time
-a always,exit -F arch=b64 -S adjtimex,settimeofday,clock_settime -k time_modification
## 6. Monitor service management and firewall modifications
-w /bin/systemctl -p x -k service_mgmt
-w /sbin/iptables -p x -k iptables_mgmt
-w /sbin/ufw -p x -k firewall_mgmt
## 7. Lock configuration (prevents root from disabling auditd; only removable via server reboot)
-e 2Key parameters to remember:
-w <path>: File or directory path to monitor (Watch).-p rwa: Permissions/actions to capture:r(read),w(write),a(attribute change like chmod/chown),x(execute).-k <key_name>: Filter key tag to easily search and parse logs later.-e 2: Sets rules into immutable mode. Even the root user cannot modify or delete rules via auditctl until the server is rebooted.
Once saved, load the rules into the kernel using:
sudo augenrules --load
sudo auditctl -lStep 3: Track Down the Real Identity Using ausearch
All audit events are logged to /var/log/audit/audit.log. Avoid using grep or cat directly, as raw audit logs are filled with encoded hex strings that are hard to decipher. The standard tool for this is ausearch.
Now, let’s search for who tampered with Nginx using the key nginx_config_changes:
sudo ausearch -k nginx_config_changes -iAdding the -i (interpret) flag is crucial so the utility automatically translates UIDs, syscall numbers, and Unix timestamps into human-readable usernames and dates. The output will look like this:
type=SYSCALL msg=audit(23/10/2026 23:15:30.124:412) : arch=x86_64 syscall=openat success=yes exit=3 a0=0xffffff9c a1=0x55d78a2f4160 a2=O_WRONLY|O_CREAT|O_TRUNC items=2 ppid=1420 pid=2105 auid=hungnv uid=root euid=root comm=nano exe=/bin/nano key="nginx_config_changes"
type=PATH msg=audit(23/10/2026 23:15:30.124:412) : item=0 name="/etc/nginx/nginx.conf" inode=131075 dev=08:01 mode=file,644 ouid=root ogid=rootPay close attention to the two most critical fields:
uid=root: The command was executed with root privileges via the/bin/nanoeditor.auid=hungnv: This is the Audit UID. It accurately records that userhungnvinitially authenticated via SSH before elevating to root withsudo su.
The culprit is unmasked—backed by solid, irrefutable evidence.
Step 4: Generate Quick Summary Reports with aureport
Every morning before starting work, you can run these three commands to get a complete security overview of the server over the last 24 hours:
## Check failed SSH login attempts (brute-force indicator)
sudo aureport -l --failed
## Summary of most accessed sensitive files
sudo aureport -f -i --summary
## Summary of root-level command executions
sudo aureport -x --summaryStep 5: Configure Log Rotation to Prevent Disk Overflow
On high-traffic servers, audit.log can grow by several gigabytes daily. To prevent disk exhaustion outages, open the configuration file /etc/audit/auditd.conf:
sudo nano /etc/audit/auditd.confAdjust the storage limits to safe thresholds:
# Limit each log file to a maximum of 50MB
max_log_file = 50
# Keep up to 10 rotated logs (caps total size at ~500MB)
num_logs = 10
# Automatically rotate logs when the size limit is reached
max_log_file_action = ROTATE
# Send warning to syslog when disk space drops below 100MB
space_left = 100
space_left_action = SYSLOG
# Suspend audit logging when free disk space falls below 50MB (avoids filling the root disk)
admin_space_left = 50
admin_space_left_action = SUSPENDSave the file and apply the new configuration with:
sudo service auditd restartTwo Critical Caveats Before Going to Production
Auditd is extremely powerful, but misconfiguring it can backfire:
- Never enable read watches (
-p r) on high-churn directories: Placing read audits on/var/log,/tmp, or application cache directories causes the kernel to generate thousands of log events per second. This spikes disk IOPS and causes immediate server degradation. - Think twice before enabling
-e 2: Once rules are locked into immutable mode, you cannot modify or delete any rule usingauditctl. Applying any changes will strictly require a full server reboot. Thoroughly test your configuration in a Staging environment for at least 2 to 3 days before enabling this on Production.
