My server was hit by an SSH brute-force attack in the middle of the night — scrolling through auth.log, I found tens of thousands of login attempts from foreign IPs, and my heart was racing. Fortunately, the attacker never got in because I had already changed the port and switched to key authentication. That incident was what finally pushed me to seriously learn intrusion detection, rather than just relying on passive defenses.
This article gets straight to the point: how to tell if your server is under attack, which tools to use for investigation, and how to respond in the right order once you’ve confirmed a compromise.
Comparing Server Intrusion Detection Methods
There are 3 main approaches, ranging from simple to complex:
Method 1: Manual Inspection
Run Linux commands manually to check system logs, running processes, and current network connections. No additional installation required — you can do this immediately when something seems off. The key point: you need to know what “normal” looks like on your server before you can spot something abnormal.
Method 2: Rootkit Scanners and Audit Tools
rkhunter, chkrootkit, Lynis — these tools automatically scan the system against known malware signatures. They catch far more than manual checks, but can generate false positives that create noise.
Method 3: Real-time IDS/Monitoring
Fail2ban, OSSEC, and Wazuh take a completely different approach: continuous 24/7 monitoring with automatic responses — blocking IPs and sending alerts as events happen. Best suited for production environments, but requires effort to set up and tune rules.
Pros and Cons Analysis
Manual Inspection
- Pros: No additional installation needed, you understand exactly what’s happening, not subject to tool false negatives
- Cons: Time-consuming, requires experience to interpret output, easy to miss things if the attacker has deleted logs or used a rootkit to hide processes
Rootkit Scanners
- Pros: Automated, catches many common rootkit types, can be scheduled to run periodically
- Cons: Cannot catch zero-days, database needs regular updates, installing packages like lwp-request or new Perl modules triggers constant false positives
Real-time IDS/Monitoring
- Pros: Immediate detection, automatic blocking, maintains a full audit trail for post-incident investigation
- Cons: Complex to set up and maintain, resource-intensive, requires tuning rules to avoid excessive false positives
Choosing the Right Method for Your Situation
Three of the most common scenarios — each requiring a different approach:
- Server suspected of being under attack right now: Start with manual inspection — the fastest way to get an overview within 5 minutes
- Want to run weekly periodic audits: Set up rkhunter or Lynis to run via cron and email the results
- Critical production server: Deploy Fail2ban at a minimum today, and consider adding OSSEC/Wazuh if you have the operational resources
In practice, I combine all three: manual inspection when an unusual alert comes in, rkhunter scanning weekly via cron, and Fail2ban running 24/7. Simple but sufficient for personal servers and small businesses.
Step-by-Step Implementation Guide
Step 1: Quick Checklist When You Suspect a Compromise
Run these commands immediately to assess the situation within minutes:
# Who is currently logged into the server?
who
w
# Recent login history — both successful and failed attempts
last -n 20
lastb -n 20
# Processes consuming abnormally high CPU/RAM
ps aux --sort=-%cpu | head -20
ps aux --sort=-%mem | head -20
# Current network connections — look for suspicious IPs
ss -tnp
# Files modified in the last 7 days in system directories
find /etc /bin /usr/bin /sbin /usr/sbin -mtime -7 -type f 2>/dev/nullRed flags to watch for: unrecognized processes with strange names, connections to foreign IPs on ports other than 80/443/22, or unknown users currently logged in.
# Check SSH logs — look for brute-force attempts and successful logins
grep "Failed password" /var/log/auth.log | tail -50
grep "Accepted" /var/log/auth.log | tail -20
# Crontab for all users — malware often installs cron backdoors
for user in $(cut -f1 -d: /etc/passwd); do
echo "=== $user ==="
crontab -u $user -l 2>/dev/null
done
# Suspicious authorized_keys — attackers often plant their own keys here
find /home /root -name "authorized_keys" -exec echo "--- {} ---" \; -exec cat {} \;Step 2: Rootkit Scanning with rkhunter and chkrootkit
# Install
apt install rkhunter chkrootkit -y # Ubuntu/Debian
yum install rkhunter chkrootkit -y # CentOS/RHEL
# Update the database before scanning
rkhunter --update
# Scan the entire system, skip step-by-step confirmations
rkhunter --check --sk
# Cross-check with chkrootkit
chkrootkitRead through any output flagged as Warning carefully — but don’t panic immediately. rkhunter frequently triggers false positives on Perl modules or /usr/bin/lwp-request. Google each specific warning before concluding you’ve been hacked.
Step 3: Comprehensive Audit with Lynis
apt install lynis -y
# Run a full system audit
lynis audit system
# Output has 3 important sections:
# - Hardening index: overall security score (higher is better)
# - Warnings: issues that need immediate attention
# - Suggestions: recommendations to improve the scoreStep 4: Isolating and Responding After Confirming a Compromise
Confirmed an intrusion? Follow this order — don’t skip any steps:
4.1. Isolate the server immediately:
# Block all inbound traffic except your IP (replace YOUR_IP)
iptables -I INPUT -s YOUR_IP -j ACCEPT
iptables -I INPUT -j DROP
# Or use ufw
ufw default deny incoming
ufw allow from YOUR_IP
ufw enable4.2. Collect evidence before cleaning up:
TIMESTAMP=$(date +%s)
ps aux > /tmp/evidence-ps-$TIMESTAMP.txt
ss -tnp > /tmp/evidence-ss-$TIMESTAMP.txt
cp /var/log/auth.log /tmp/evidence-auth-$TIMESTAMP.log
crontab -l > /tmp/evidence-cron-root-$TIMESTAMP.txt4.3. Kill malicious processes and remove backdoors:
# Kill by PID (get it from ps aux output above)
kill -9 SUSPICIOUS_PROCESS_PID
# Change passwords for all users with shell login
grep -v '/nologin\|/false' /etc/passwd
passwd root
passwd other_username
# Lock unused user accounts
usermod -L unused_username
# Remove suspicious SSH keys — open each file and delete unrecognized lines
nano /root/.ssh/authorized_keys
nano /home/your_user/.ssh/authorized_keysStep 5: Setting Up Fail2ban for Future Defense
apt install fail2ban -y
# Create a local config (don't modify the original .conf to avoid it being overwritten on updates)
cat > /etc/fail2ban/jail.local << 'EOF'
[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 5
[sshd]
enabled = true
port = ssh
EOF
systemctl enable fail2ban
systemctl start fail2ban
# Check currently banned IPs
fail2ban-client status sshdAfter enabling Fail2ban on my server, I see roughly 50–150 IPs auto-banned every day just for SSH brute-forcing — all bots, with no real users ever affected. Beyond Fail2ban, I always do these 2 things right after setting up a new server:
# Disable SSH password login — allow key authentication only
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd
# Change the default SSH port (eliminates ~90% of noise from auto-scanners)
sed -i 's/^#*Port.*/Port 2222/' /etc/ssh/sshd_config
systemctl restart sshdPractical Takeaways
Hackers aren’t targeting your server specifically — they use automated scanners to sweep millions of IPs at once. Servers get compromised primarily because they haven’t been hardened, not because someone singled them out. That night I spent dealing with an SSH brute-force at 2 AM taught me a very practical lesson: setting up security upfront takes 30 minutes; putting out fires after an incident can cost you a full day and your data.
My current workflow: Fail2ban running in the background 24/7, rkhunter scanning weekly via cron. When I get an unusual alert, I run through the Step 1 checklist before doing anything else. It’s enough for most personal servers and small businesses without needing complex infrastructure.
