Hello everyone! As an IT engineer, I want to share an important lesson: Linux security log auditing. Often, we mistakenly believe that simply installing a firewall or updating software is enough.
However, the reality is not that simple. I once had to wake up in the middle of the night to deal with an SSH brute-force attack on my server – an unpleasant experience that I don’t want anyone else to go through. From that incident, I realized a crucial point: security needs to be established and checked from the very beginning, and log auditing is the foundation.
Security logs can be imagined as a surveillance camera system and a detailed activity log of the server. They record everything: who logged in, when, which files were accessed, system errors, and abnormal behavior. Mastering how to read and analyze these logs will help you detect attacks early, understand system status, and proactively protect your server effectively.
This article will be your guide to performing Linux security log auditing, from the most basic operations for beginners to advanced techniques. We will explore “why” and “how” to effectively leverage the important data that your server constantly records.
Quick Start: Begin log auditing in 5 minutes
Want to quickly check your server’s security status right now? Here are some basic commands you can use.
1. Check recent logins
Want to know who logged into your server recently, from where, and when? The last command will display this list. If you see an unfamiliar user or IP, pay special attention.
last
2. View failed SSH login attempts
If your server is currently a target of an SSH brute-force attack, you will see a large number of failed login records. Quickly check using the following command:
grep "Failed password" /var/log/auth.log | tail -n 20
This command filters the 20 most recent lines in /var/log/auth.log (or /var/log/secure on RHEL/CentOS) searching for the string “Failed password”. Many alerts from the same IP are a clear sign of a brute-force attack.
3. Check general system logs with journalctl
journalctl is an effective log management tool, especially on systems using systemd. It aggregates logs from multiple sources.
journalctl -xe --since "1 hour ago"
The above command will display log events, including errors and warnings, from one hour ago to the present. The -xe option provides more detailed information.
With just these three simple commands, you can get a preliminary grasp of your server’s security situation. Now, let’s explore in more depth.
Detailed Explanation: What are Security Logs and Their Importance?
Security logs are not merely text files; they are valuable data sources. This information helps us understand server activity, quickly detect threats, and ensure compliance with necessary security regulations.
What are Security Logs and Their Importance?
Security logs record important security-related events on Linux, such as:
- Login/Logout: Records who, from where, and when someone logged in or out, and whether the action was successful or failed.
- Sudo Privilege Usage: Tracks users who have executed commands with root privileges.
- Configuration Changes: Includes software installation/removal, as well as changes to file access permissions.
- System Errors: Notifications about hardware, software issues, out-of-memory conditions, or signs of DoS attacks.
Log auditing offers many practical benefits:
- Early Intrusion Detection: Helps identify unusual activities such as continuous failed logins or access to sensitive files.
- Incident Investigation: Logs provide crucial evidence to determine the cause, timing, and method of an attack.
- Regulatory Compliance: Many security standards (such as PCI DSS, HIPAA) require logging and reviewing system logs to ensure compliance.
Key Security Log Files and How to Read Them
The Linux system has many log files, each serving a specific purpose. Here are the most important files you need to know:
/var/log/auth.log(Debian/Ubuntu) or/var/log/secure(RHEL/CentOS): Records authentication and authorization events, including logins, sudo usage, and SSH connections./var/log/syslog(Debian/Ubuntu) or/var/log/messages(RHEL/CentOS): Contains general system messages, warnings, and errors from various services.journalctl: A powerful tool on systemd-based systems for reading and managing logs. It aggregates logs from multiple sources, providing a comprehensive view of system activity.
Basic Log Reading and Analysis
To view the latest log lines in real-time, use tail -f:
tail -f /var/log/auth.log
To view the entire content of a large log file (use less for easy scrolling and searching):
less /var/log/syslog
Use grep to filter specific information from logs:
grep "authentication failure" /var/log/auth.log
grep -i "error" /var/log/syslog -C 5
With journalctl, you can flexibly filter logs by service, PID, or priority level:
journalctl -u ssh.service --since "2024-03-10 10:00:00"
journalctl -p err..crit
Advanced: Linux Audit System (auditd) and Integration
When security requirements are more complex or strict compliance standards need to be met, the Linux Audit Daemon (auditd) is a powerful solution. This system provides the ability to record detailed kernel-level security events, making log data very difficult to tamper with.
1. Introduction and Installation of auditd
auditd can monitor almost all critical activities on the system: from accessing sensitive files/directories, program execution, to permission changes. To install:
On Debian/Ubuntu:
sudo apt update
sudo apt install auditd audispd-plugins
sudo systemctl enable auditd --now
On RHEL/CentOS:
sudo yum install audit
sudo systemctl enable auditd --now
2. Configure Audit Rules
auditd rules are defined in /etc/audit/audit.rules or in the /etc/audit/rules.d/ directory. You can add them using the auditctl command.
Example: Monitoring Access to the /etc/passwd file
To monitor all read, write, or attribute change activities on the /etc/passwd file:
sudo auditctl -w /etc/passwd -p wra -k passwd_changes
sudo auditctl -D # Delete all existing rules
sudo auditctl -R /etc/audit/audit.rules # Reload rules from file
The above rule monitors write, read, and attribute change activities on the /etc/passwd file, assigning them a key called passwd_changes for easier searching later.
3. Analyze auditd Logs with ausearch and aureport
auditd logs are written to /var/log/audit/audit.log and require specialized tools for reading and analysis.
Searching with ausearch
To search for events related to the passwd_changes key that we configured:
sudo ausearch -k passwd_changes
Generating Reports with aureport
To generate a summary report, for example, about failed login attempts:
sudo aureport --failed --login
4. Integration with Centralized Systems
When the volume of logs becomes too large, integrating with a centralized management system is necessary. You can use rsyslog/syslog-ng to forward logs to a separate server, or more powerful solutions like Elastic Stack (ELK) or Grafana Loki. These systems help collect, analyze, and visualize log data effectively.
Practical Tips: Maintaining Security and Efficiency
Log auditing is an ongoing process that requires constant attention to ensure system security.
1. Establish a Periodic Log Review Process
Dedicate time daily or weekly to review important logs. Use simple scripts to automate this, for example, sending email reports of suspicious events. This helps you detect any potential issues early.
2. Understand What “Normal” Means
To detect anomalies, you first need to understand what “normal” looks like. Monitor logs when the system is stable to grasp daily activity patterns. Any significant deviations thereafter will be easily recognized.
3. Manage Log Volume
Logs can quickly consume disk space if not managed. Configure logrotate (available on most Linux distributions) to automatically compress, rotate, and delete old log files. This ensures disk space and prevents the loss of important logs due to a full disk.
sudo less /etc/logrotate.conf
4. Automated Alerts for Critical Events
Instead of continuously checking logs, set up automated alerts. Tools like Logwatch can summarize and send reports via email. For extremely critical events (e.g., root login from an unknown IP), you can use a small script combined with grep and mail to send instant alerts.
#!/bin/bash
LOG_FILE="/var/log/auth.log"
ALERT_EMAIL="[email protected]"
# Find failed root login attempts in the last 5 minutes
FAILED_ROOT_LOGINS=$(grep "authentication failure.*ruser=root" $LOG_FILE | awk -v date="$(date --date='5 minutes ago' +'%b %e %H:%M')" '$0 ~ date {print}')
if [ -n "$FAILED_ROOT_LOGINS" ]; then
echo "ALERT: Failed root login detected in the last 5 minutes!" | mail -s "Linux Security Alert: Failed Root Login" "$ALERT_EMAIL"
echo "$FAILED_ROOT_LOGINS" | mail -s "Details of Failed Root Login" "$ALERT_EMAIL"
fi
You can set this script to run periodically using cron for continuous monitoring.
5. Don’t Rely on a Single Layer of Protection
Audit logs are a crucial part, but not the sole protection measure. Combine them with other security layers such as firewalls (ufw, iptables), using SSH keys instead of passwords, regular software updates, and deploying fail2ban to block brute-force attacks from malicious IPs.
Conclusion
Linux security log auditing can be complex at first, but it is an extremely valuable skill for anyone working with systems. Remembering that night I had to wake up to deal with an SSH brute-force incident, I clearly see the importance of being proactive. Taking the time to learn, set up audit rules, and regularly check logs will make you much more confident in protecting your server from threats.
Think of logs as silent sentinels, always recording everything so you can act promptly. Don’t wait until an incident occurs to start learning. Start today to make your server more secure!
