Guide to Using YARA Rules on Linux: Writing Custom Rules and Automating Webshell Scans

Security tutorial - IT technology blog
Security tutorial - IT technology blog

3 Approaches to Scanning for Malware on Linux

When managing production server clusters, detecting rogue files typically comes down to three approaches:

  • Checksum Verification (MD5/SHA256): Compare current file hashes against clean builds from CI/CD or package managers (such as debsums on Ubuntu). A modified hash indicates tampering.
  • Traditional Antivirus (such as ClamAV): Automatically scans files against existing signature databases provided by security vendors.
  • Custom Pattern Matching (YARA Rules): Write custom rules matching text strings, regular expressions, and binary bytes with logical conditions to hunt down the exact webshells targeting your codebase.

Practical Comparison of the Three Approaches

Each tool excels in a distinct scenario:

1. Checksum Comparison

  • Pros: Extremely fast, consuming less than 1% CPU. Virtually zero false positives on static system binaries.
  • Cons: Useless against newly created files in temporary writable directories like /tmp or /dev/shm. An attacker only needs to append a single space to a PHP file to completely change its SHA256 hash.

2. Traditional Antivirus (ClamAV)

  • Pros: Massive signature database. Easy to use for system administration beginners.
  • Cons: The clamd daemon consumes 1.2 GB to 1.8 GB of RAM just to load its database. Running it on a 1 GB–2 GB VPS frequently triggers Out-of-Memory (OOM) errors. In addition, ClamAV updates signatures relatively slowly against custom-obfuscated webshell variants.

3. YARA Rules

  • Pros: Lightning-fast byte scanning powered by the Aho-Corasick algorithm. Memory usage rarely exceeds 50 MB of RAM. You can proactively write rules to catch PHP webshells, cryptomining bash scripts, or masqueraded ELF binaries.
  • Cons: Requires a solid understanding of malicious string patterns. Overly broad conditions cause false positives that can flood your notification inbox.

Why DevOps Engineers Should Incorporate YARA into Operations

In the middle of the night on a weekend, my VPS was hit by an SSH port scan followed by a backdoor infection. At 2 AM, while sifting through line after line in auth.log, I had to manually inspect recently modified files in /var/www/uploads—an attack vector that could otherwise be mitigated by deploying Fail2ban to defend against SSH brute force. It was a nerve-wracking experience because I had no idea whether the attacker had planted additional scripts deeper inside the system.

That incident proved that relying solely on off-the-shelf antivirus is not enough. Today’s attackers routinely obfuscate strings with multi-layer base64 encoding or disguise binary names as system processes like [kworker/0:1]. YARA tackles this cleanly: it works like grep on steroids with logical conditions and binary inspection capabilities—a core skill when hunting malware on Linux with YARA rules. You can integrate YARA into CI/CD pipelines or schedule daily cron jobs without bottlenecking the server’s CPU.

Step-by-Step Implementation of YARA on Linux

Step 1: Install YARA

On Ubuntu or Debian, install it directly via APT:

sudo apt update
sudo apt install -y yara

# Verify version
yara --version

If your project requires extension modules such as ELF header parsing or automated hashing, you can build it from source via the official VirusTotal GitHub repository.

Step 2: Understand the Basic Structure of a Rule File

Each YARA file (with a .yar extension) consists of three sections: meta (contains metadata/notes), strings (declares text, hex, or regex patterns to look for), and condition (logical expressions determining whether a file matches).

rule Detect_Simple_PHP_Backdoor
{
    meta:
        author = "SysAdmin"
        description = "Detects dangerous eval combined with base64 decode"
        date = "2026-11-19"

    strings:
        $php = "<?php"
        $eval = "eval("
        $base64 = "base64_decode("
        $system_call = /system\s*\(/ nocase

    condition:
        $php at 0 and (($eval and $base64) or $system_call)
}

Step 3: Write Rules to Detect PHP Webshells and Hidden Shell Commands

Attackers often invoke shell execution functions directly or embed hex-encoded strings to bypass WAFs—such as an open-source WAF built with ModSecurity and OWASP CRS. We can compile these indicators into /etc/yara/webshell_rules.yar:

sudo mkdir -p /etc/yara
sudo nano /etc/yara/webshell_rules.yar

The rule content inspecting file sizes and sensitive functions:

rule Linux_PHP_Suspicious_Webshell
{
    meta:
        description = "Detects PHP webshell invoking shell commands or using hex payloads"
        severity = "High"

    strings:
        $tag = "<?php"
        $f1 = "shell_exec"
        $f2 = "passthru"
        $f3 = "popen"
        $f4 = "proc_open"
        $f5 = "assert"
        
        // Hex payload representing: eval(base64
        $hex_payload = { 65 76 61 6c 28 62 61 73 65 36 34 }
        $c99_pattern = "c99shell" nocase

    condition:
        $tag and (2 of ($f1, $f2, $f3, $f4, $f5) or $hex_payload or $c99_pattern) and filesize < 2MB
}

Step 4: Run Real-World Scans on Web Directories

After saving the rules, scan high-risk directories such as /var/www/html or /tmp using the yara command:

# Recursive scan (-r), print matched strings (-s), disable warnings (-w)
yara -r -s -w /etc/yara/webshell_rules.yar /var/www/html

# Quick scan on /tmp to find suspicious executable files
yara -r -w /etc/yara/webshell_rules.yar /tmp

If suspicious files are detected, the terminal outputs the rule name along with the offset and matching string content:

Linux_PHP_Suspicious_Webshell /var/www/html/uploads/avatar_shell.php
0x1a:$f1: shell_exec
0x42:$f2: passthru

Step 5: Schedule a Cron Job for Automated Scanning and Logging

Create a shell script at /usr/local/bin/yara-auto-scan.sh for periodic scans:

#!/usr/bin/env bash
set -euo pipefail

RULE_FILE="/etc/yara/webshell_rules.yar"
SCAN_DIRS=("/var/www/html/uploads" "/tmp" "/dev/shm")
LOG_FILE="/var/log/yara-scan.log"
DATE=$(date '+%Y-%m-%d %H:%M:%S')

echo "[$DATE] Starting YARA scan..." >> "$LOG_FILE"

for DIR in "${SCAN_DIRS[@]}"; do
    if [ -d "$DIR" ]; then
        OUTPUT=$(yara -r -w "$RULE_FILE" "$DIR" 2>/dev/null || true)
        if [ -n "$OUTPUT" ]; then
            echo "[$DATE] WARNING: Suspicious file detected in $DIR:" >> "$LOG_FILE"
            echo "$OUTPUT" >> "$LOG_FILE"
            # Telegram/Slack bot alerts via curl can be integrated here
        fi
    fi
done

echo "[$DATE] Scan completed." >> "$LOG_FILE"

Grant execution permissions and add to the system crontab:

sudo chmod +x /usr/local/bin/yara-auto-scan.sh

# Open root crontab
sudo crontab -e

# Run every 3 hours:
0 */3 * * * /usr/local/bin/yara-auto-scan.sh

Once this cron job is configured, any rogue files uploaded to web directories or dropped into temp folders will be immediately flagged with a clear audit trail, complementing routine Linux server security auditing and hardening.

Share: