Last July, I discovered that /etc/sudoers on the staging server had been modified. No notification, no email, nothing — just a random permission error during an afternoon deploy. It took nearly two hours of backtracking to find out who made the change, when, and from which terminal. That was when I started taking auditd seriously.
The Real Problem: When syslog Can’t Answer “Who Did That?”
I’ve been using Fedora as my primary development machine for two years and have been pretty happy with the package update cadence. But when I brought Fedora Server into production, I quickly realized that journald and syslog simply weren’t designed to answer questions like:
- Which process read
/etc/passwdat 3 AM? - Who ran what
sudocommands yesterday? - Has the
/usr/bin/python3binary been replaced?
Journald logs by service, not by file. You can tell that the SSH daemon established a connection, but you can’t tell whether that user then ran cat /etc/shadow. This is a dangerous blind spot — especially when you need to demonstrate compliance with CIS Benchmark or STIG.
SELinux handles access control, while auditd handles recording evidence — two entirely different things that complement each other rather than replace one another.
Root Cause Analysis: Why Audit at the Kernel Level?
The Linux Audit System operates through the Linux Audit Framework — a subsystem built into the kernel since version 2.6. Unlike syslog, which only records what applications actively report, audit hooks directly into syscalls:
open(),read(),write()— file read/writeexecve()— command executionsetuid(),setgid()— privilege changes
Because it hooks at the syscall level, there’s no way to bypass it — even root cannot perform an operation without leaving a trace, as long as auditd is running and rules are configured correctly.
On Fedora, auditd is typically pre-installed. If it isn’t:
sudo dnf install audit audit-libs -y
sudo systemctl enable --now auditd
# Check status
sudo auditctl -sApproaches: From Ad-hoc to Production-Grade
Approach 1: Use auditctl for Temporary Rules
If you just need to debug quickly, auditctl lets you add rules immediately without restarting the service:
# Monitor all write operations on /etc/passwd
sudo auditctl -w /etc/passwd -p wa -k passwd_watch
# View currently active rules
sudo auditctl -l
# Remove all temporary rules
sudo auditctl -DFlags -p wa: monitor write (w) and attribute change (a — chmod, chown). Key -k passwd_watch is a label for filtering logs later.
Limitation: Rules are lost after reboot. Only suitable for debugging, not for production.
Approach 2: Persistent Rules in /etc/audit/rules.d/
Fedora loads persistent rules from /etc/audit/rules.d/ in alphabetical order when auditd starts. I split them by function for easier management:
sudo nano /etc/audit/rules.d/10-identity.rules## Monitor user identity files
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/gshadow -p wa -k identity
## Monitor sudoers
-w /etc/sudoers -p wa -k sudoers_change
-w /etc/sudoers.d/ -p wa -k sudoers_change
## SSH config
-w /etc/ssh/sshd_config -p wa -k sshd_configsudo nano /etc/audit/rules.d/20-priv-commands.rules## Track privileged commands by real users (auid >= 1000, exclude kernel processes)
-a always,exit -F path=/usr/bin/sudo -F perm=x -F auid>=1000 -F auid!=4294967295 -k priv_cmd
-a always,exit -F path=/usr/bin/su -F perm=x -F auid>=1000 -F auid!=4294967295 -k priv_cmd
-a always,exit -F path=/usr/bin/newgrp -F perm=x -F auid>=1000 -F auid!=4294967295 -k priv_cmd
## Monitor kernel module changes (rootkit indicator)
-w /sbin/insmod -p x -k module_insertion
-w /sbin/rmmod -p x -k module_insertion
-w /sbin/modprobe -p x -k module_insertionAfter writing the rules, check syntax and reload:
sudo augenrules --check
sudo augenrules --load
# On Fedora: do NOT use systemctl restart auditd
# auditd has its own signal handling — use this instead:
sudo service auditd restartApproach 3: Use SCAP/STIG Presets for Compliance
If you need to meet compliance requirements, there’s no need to write rules from scratch. Fedora supports OpenSCAP:
sudo dnf install openscap-scanner scap-security-guide -y
# List available profiles
sudo oscap info /usr/share/xml/scap/ssg/content/ssg-fedora-ds.xml | grep profile
# Dry-run scan (no system changes)
sudo oscap xccdf eval \
--profile xccdf_org.ssgproject.content_profile_ospp \
--results /tmp/scan-results.xml \
/usr/share/xml/scap/ssg/content/ssg-fedora-ds.xmlThe OSPP profile generates a complete set of audit rules to standard. The downside: more rules than you may need, which can cause audit logs to grow quickly if not tuned.
The Best Approach: Minimal Ruleset + Effective Log Analysis
After six months in production, I’ve found the most effective approach is: start minimal, expand based on real needs. Too many rules → log flood → nobody reads them → pointless.
Add a base file to control the buffer:
sudo nano /etc/audit/rules.d/00-base.rules## Delete existing rules
-D
## Buffer size (increase if you see audit: backlog limit exceeded in dmesg)
-b 8192
## Failure mode: 1=log via printk, 2=kernel panic (use 2 only for strict compliance)
-f 1Configure rotation in /etc/audit/auditd.conf to prevent logs from eating up disk space:
max_log_file = 50 # MB per file
num_logs = 10 # Keep 10 rotated files (~500MB total)
max_log_file_action = ROTATE
space_left = 500 # Warn when 500MB disk space remains
space_left_action = SYSLOG
admin_space_left = 50
admin_space_left_action = SUSPEND # Suspend audit when disk is nearly fullAnalyzing Logs with ausearch and aureport
Raw logs at /var/log/audit/audit.log aren’t meant for human eyes. Use the bundled tools instead:
# Find all events related to the "identity" key from today
sudo ausearch -k identity --start today
# Find all sudo commands by user alice (-i flag shows username instead of UID)
sudo ausearch -k priv_cmd --uid alice -i
# See who read/wrote /etc/shadow this month
sudo ausearch -f /etc/shadow --start this-month -i
# Full summary report
sudo aureport --summary
# Authentication report (login/logout/sudo)
sudo aureport -au
# Failed events only
sudo aureport --failed -auA trick I use during investigations: combine ausearch with --raw and pipe to audisp to see the full context of an event:
# View event by specific event ID
sudo ausearch -a 1234567 -i
# Export to file for offline analysis
sudo ausearch -k sudoers_change --start last-week > /tmp/sudoers-audit.txtLessons Learned After Six Months in Production
First, the auid!=4294967295 filter (the unset value, meaning processes started by the kernel before any user logs in) is critical. Without it, logs fill up with events from systemd services that are irrelevant and make it hard to find what you actually need.
Second, don’t enable -e 2 (immutable mode) until you’re confident the ruleset is correct. This mode locks the audit config — only a reboot can undo it. I once lost 20 minutes because I added that line too early.
Third, combine auditd with logwatch or a simple SIEM to receive daily email reports. Having logs you never read is as good as having no logs at all. This cron job gets the job done adequately:
## Add to crontab: send audit report at 7 AM
0 7 * * * /usr/sbin/aureport --summary | mail -s "[Audit Report] $(hostname) $(date +%F)" [email protected]If you’re managing a Fedora Server and haven’t configured auditd yet, this should be done before hardening SSH or setting up firewall rules — because without audit logs, you’ll never know whether your other security measures are actually working, and more importantly, you’ll have no evidence to investigate when an incident occurs.
