When the ls Command Suddenly Betrays You
It was 2 AM, and my phone buzzed incessantly on the desk. The monitoring system reported that the Nginx Production server running Ubuntu 22.04 was constantly returning 500 errors. I SSH’d in to check; Nginx logs were spotless, and the config files hadn’t changed compared to the Git backup. However, when I typed ls -l /usr/sbin/nginx, I froze. The binary file size was off by about 12KB compared to the standard build in the Staging environment.
A nightmare scenario unfolded: the server had been compromised. An attacker might have replaced the executable with a backdoored version, leaving the system still vulnerable after updating. Basic commands like ls or stat were no longer trustworthy. To be certain whether the files on the disk were still original, I needed a tool to verify checksums against the official Ubuntu repositories.
3 Reasons Why System Files Get “Corrupted”
During my time in operations, I’ve identified three main reasons why system binaries or libraries get modified:
- Physical Errors (Bit Rot): Data on the hard drive or RAM is written incorrectly due to hardware aging. While the frequency is low, it can definitely happen to servers running continuously for years.
- Human Error: A colleague with sudo privileges might have accidentally used
vimto directly edit a file in/usr/binfor a quick debug and then forgot to roll it back. - Security Incidents: Hackers replace management tools like
ps,netstat, orsshdwith malware to hide processes and steal data.
Don’t rush to format and reinstall the OS immediately. That takes hours of reconfiguring from scratch, unlike the resilience found when deploying Ubuntu Core. We need concrete evidence to make the right decision.
Debsums: The “Microscope” for System Errors
Many might think of manually comparing MD5/SHA256 hashes for each file. This is impossible since a standard Ubuntu system has over 50,000 files in /usr. Some use dpkg -V, but its output is quite cluttered and hard to filter.
Debsums excels by directly comparing files on disk with the standard MD5 database at /var/lib/dpkg/info/*.md5sums. If there is even a 1-bit discrepancy, it alerts you immediately.
1. Installing debsums
This tool is not included by default. You can install it quickly via apt:
sudo apt update
sudo apt install debsums -y2. Quickly Scanning for Modified Files
To find executables that no longer match the original, use the command:
sudo debsums -cThe -c (changed) flag helps you filter out normal files. If the command returns no results, your system is still safe. If a list of files appears, those are the locations that need immediate review.
3. Checking Configuration Files
By default, debsums skips the /etc directory because admins frequently modify config files there. However, if you suspect someone has injected a malicious configuration, use the -a (all) flag:
sudo debsums -ca4. Checking a Specific Package
Back to my Nginx case. Instead of scanning the entire disk, I focused only on the Nginx package to save time:
sudo debsums nginxIf the result shows FAILED as seen below, the binary file has definitely been tampered with:
/usr/sbin/nginx FAILEDRestoring System Files to a Clean State
Once a corrupted or hacked file is identified, the safest way is to have apt redownload the package and overwrite the current files. This process is as critical as automating Ubuntu security patches to maintain a healthy environment. Suppose the nginx-core package reports an error; run the command:
sudo apt install --reinstall nginx-coreAfter reinstalling, run debsums again. If the result is OK, your files are clean.
Pro Tip: Automate to Sleep Soundly
Don’t wait for a server incident to check. I usually set up a small script to run via cron weekly. This script automatically scans the system and sends an alert to Telegram if a FAILED file is detected.
#!/bin/bash
# Check for modified system files
LOG_FILE="/var/log/debsums_check.log"
RESULTS=$(debsums -c 2>&1)
if [ ! -z "$RESULTS" ]; then
echo "Warning: Modified files detected on $(date)" > $LOG_FILE
echo "$RESULTS" >> $LOG_FILE
# Add curl command to send Telegram notification here
fiImportant Note: Debsums is not a “silver bullet.” If a hacker gains root access for long enough, they can modify the checksum database in /var/lib/dpkg/info/. In that case, you need to compare against another clean server or use offline check tools.
Summary
Professional server administration isn’t about guesswork. Debsums provides you with concrete figures and evidence regarding system integrity. Try running sudo debsums -c today. You might discover some “interesting” changes you never knew existed.
