A 2 AM Nightmare and Hard-earned Lessons in DFIR
Not long ago, I was jolted awake in the middle of the night by a monitoring system screaming in red. A production server cluster was being relentlessly brute-forced via SSH, and worse, the intruders had managed to get inside. My heart was pounding faster than the cooling fans in the server room. The problem wasn’t just one machine; it was dozens of servers running on the same blueprint.
I needed to know immediately: What commands did the attacker run? Where is the backdoor hidden? How many machines are compromised? If I had to SSH into each one to check history, audit auth.log, or hunt for suspicious files, the system would be in ruins by morning. From that painful experience, I realized that a centralized Incident Response (DFIR) tool is indispensable.
Approaches to Digital Evidence Collection
1. Manual “Brute Force” (SSH + Scripts)
This is the most common method when starting out: writing a bash script using ps, netstat, and lsof, then pushing it to servers via SCP to execute.
- Pros: No prior preparation required.
- Cons: Extremely slow at scale. Most dangerously, digital evidence can be easily tampered with if the attacker has gained root access.
2. Using SIEM or Log Management (Wazuh, ELK)
Wazuh or ELK are excellent for real-time monitoring with visual dashboards.
- Pros: Centralized logs, easy to track trends.
- Cons: Deep querying of specific files on disk or memory dumping for malware analysis isn’t usually a SIEM’s strength. Additionally, the massive cost of log storage can be a barrier.
3. Velociraptor – Endpoint Visibility & Forensics
Velociraptor was born for professional Incident Response. It allows you to perform deep system queries using VQL (Velociraptor Query Language).
- Pros: Ultra-lightweight agent (consuming only 20-30MB RAM). You can command 500 servers simultaneously: “Find files with this hash” and get results in under a minute.
- Cons: Requires time to learn VQL, but the efficiency gained is well worth the investment.
Why Velociraptor is the Top Choice for Linux?
I chose Velociraptor for its near-instant response. When something goes down, I need answers immediately, not after waiting for logs to sync to a SIEM. This tool enables infrastructure-wide “Hunting” without crashing production servers.
Guide to Deploying Velociraptor Server
First, set up a central management server. An Ubuntu 22.04 VPS with 2 vCPUs and 4GB RAM is enough to manage about 500-1000 agents.
Step 1: Download the Binary
Grab the latest version from GitHub. Here, I’m using the linux-amd64 build:
wget https://github.com/Velocidex/velociraptor/releases/download/v0.72/velociraptor-v0.72-linux-amd64
chmod +x velociraptor-v0.72-linux-amd64
sudo mv velociraptor-v0.72-linux-amd64 /usr/local/bin/velociraptorStep 2: System Configuration
Run the following command to start the configuration wizard:
velociraptor config generate -iPay attention to these parameters: Data storage path, UI port (default 8889), and Agent port (default 8006). Once finished, you’ll have two files: server.config.yaml (keep this secure) and client.config.yaml (used for agent installation).
Step 3: Set up Systemd Service
To ensure the server starts automatically on boot, create a service file:
sudo nano /etc/systemd/system/velociraptor.serviceThe basic service file content looks like this:
[Unit]
Description=Velociraptor Linux Server
After=network.target
[Service]
ExecStart=/usr/local/bin/velociraptor --config /etc/velociraptor/server.config.yaml server
Restart=always
[Install]
WantedBy=multi-user.targetEnable and start the service with: sudo systemctl enable --now velociraptor.
Installing the Agent on Target Servers
Transfer the client.config.yaml file to the servers you want to monitor. Instead of doing it manually, use Ansible to deploy it across the fleet in just 5 minutes.
Quick Installation:
On the target machine, after downloading the binary, simply run:
sudo /usr/local/bin/velociraptor --config client.config.yaml client installThe agent will automatically register with the Dashboard. You’ll see the server list populate immediately.
In Action: Tracing SSH Brute-force
Suppose you suspect someone is brute-forcing passwords. Instead of manually parsing log files, use VQL to filter the top 10 attacking IPs:
SELECT Count, User, IP FROM source(artifact="Linux.Ssh.AuthLogs")
WHERE Message =~ "Failed password"
GROUP BY IP
ORDER BY Count DESC
LIMIT 10In just 2 seconds, you’ll have a “blacklist” and can block those IPs on your firewall. If you want to check the /etc/passwd file for suspicious users across 100 servers, simply create a new Hunt and select the Linux.Sys.Users artifact. The results will be aggregated into a single table for easy auditing.
Final Thoughts from Real-world Experience
Don’t wait until the “house is on fire” to look for a hose. Pre-installing Velociraptor consumes negligible resources (nearly 0% CPU when idle), but it’s a lifesaver during an incident. Instead of panicking, just open your Dashboard, write a few lines of VQL, and take control of the situation. Wishing you all peaceful nights, free from sudden 2 AM wake-up calls!
