When System Tools Are Deceived by Attackers
It’s 2 AM, and the monitoring system triggers a red alert: a critical database server is suddenly pushing outbound traffic up to 500Mbps. I log in to check using top, ps -aux, and netstat, but everything looks suspiciously clean. No process is hogging the CPU, and no strange connections appear to be active.
After a thorough review, I start to realize the problem: the server has been infected with a Rootkit. When malware gains control of the Kernel, it modifies the output of system commands. What you see on the terminal is nothing more than a carefully staged performance.
To unmask the hidden intruder, Memory Forensics (digital investigation on RAM) is the only way forward. RAM doesn’t lie. Every process, network connection, or piece of executable code leaves a trace here. And the Volatility Framework is the most powerful microscope for examining those traces.
Three Incident Investigation Methods: Which Weapon to Choose?
When you suspect a server has been compromised, you usually face three choices:
1. Live Analysis
You use lsof, ps directly on the server. This is fast but extremely risky because Rootkits can hide files and processes. You are essentially asking the thief: “Are you in the house?”
2. Disk Forensics
This method helps find executable files or deleted logs. However, it is completely helpless against “Fileless Malware” – malware that resides only in RAM and disappears as soon as the server reboots.
3. Memory Forensics (RAM Analysis with Volatility)
This involves capturing the entire RAM data for offline analysis. You will see hidden processes, code injected into clean processes, and even encryption keys. Although it takes more time, it is the only way to expose sophisticated attacks and perform deep malware analysis.
Hands-on: Malware Hunting Workflow with Volatility 3
Volatility 3 is now the new standard with a modern architecture that no longer requires manual Profile creation like the old version. Here is the actual workflow I typically apply.
Step 1: Collecting a Memory Dump with AVML
Never install Volatility on the infected server. Use Microsoft’s AVML to capture the RAM. This tool is lightweight and doesn’t require complex kernel module building.
# Download AVML directly to the target server
wget https://github.com/microsoft/avml/releases/download/v0.13.0/avml
chmod +x avml
# Dump RAM to a file (e.g., if the server has 16GB RAM, the dump file will be roughly the same size)
sudo ./avml memory.dmpAfter that, transfer the memory.dmp file to a separate analysis machine via SCP or an external drive.
Step 2: Setting Up the Analysis Environment
On your analysis machine (Linux or macOS), you need to prepare Python 3:
git clone https://github.com/volatilityfoundation/volatility3.git
cd volatility3
pip3 install -r requirements.txtStep 3: Unmasking Hidden Processes (Process Analysis)
The linux.pslist command only lists processes according to the Kernel’s list. To find Rootkits, use linux.psscan. This plugin scans the entire memory for process structures, including those that have been unlinked from the operating system’s management list.
# Find hidden or terminated processes
python3 vol.py -f memory.dmp linux.psscanSuspicious signs: Look for processes with system-like names (such as kworker) but with a Parent Process ID (PPID) pointing to a shell or an unrelated process.
Step 4: Tracing Hidden Network Connections
If netstat on the server shows nothing, Volatility will show you the truth through its netstat plugin.
python3 vol.py -f memory.dmp linux.netstatPay attention to ESTABLISHED connections to strange IPs or “sensitive” ports like 4444, 8080, or 1337. A connection from nginx to an unknown foreign IP is a major red flag, and identifying these is the first step to stop sneaky connections that bypass traditional tools.
Step 5: Detecting Memory Injection Techniques
Many modern malwares do not run as standalone files. They inject malicious code into trusted processes like sshd. The malfind plugin searches for memory regions with RWX (Read-Write-Execute) permissions – a typical characteristic of shellcode.
python3 vol.py -f memory.dmp linux.malfindIf you see code snippets starting with the hex \xeb or \x31, it is almost certainly shellcode waiting for commands from an attacker.
Important Note on Symbol Tables
Linux forensics is more challenging than Windows because the Linux Kernel changes constantly. Volatility 3 requires a Symbol Table (in JSON format) to understand the memory structure of a specific Kernel version.
If the tool doesn’t recognize the dump file, you need to run uname -r on the target server. Then, find and download the corresponding debug symbols package from the distro’s repository (Ubuntu, CentOS…) to generate the Symbol set for Volatility.
Digital forensics is a high-pressure battle of wits, especially during an active incident. However, with Volatility, you no longer have to guess. Every intrusion behavior, no matter how sophisticated, must eventually come to light under the lens of RAM Forensics, providing a level of detail comparable to tracking every change on Linux with advanced auditing tools.
