Installing ModSecurity v3 and OWASP CRS on Nginx Ubuntu: Building a Production-Ready Web Server WAF - ITFROMZERO - Share tobe shared! UFW and Fail2ban cannot inspect malicious payloads inside HTTP requests. Here is a step-by-step guide to deploying ModSecurity v3 with OWASP CRS on Nginx in Ubuntu to build a Layer 7 WAF that blocks SQLi, XSS, and RCE.

Table of Contents

1. The Reality: Server Compromised Despite Active UFW and Fail2ban

Many Ubuntu sysadmins share a common habit: configure UFW to close unused ports, set up Fail2ban to block SSH brute-force attempts, and assume everything is secure. In reality, production environments are rarely that simple.

Recently, one of our web applications had its entire users table dumped. Inspecting the logs showed UFW running smoothly and Fail2ban completely silent. The attacker never flooded the server with thousands of brute-force requests to trigger rate-limiting thresholds. Instead, they sent just three HTTP requests containing a SQL injection exploit payload:

' UNION SELECT id, username, password_hash FROM admin_users--

Nginx received the packets over port 443 just like any legitimate traffic. The PHP-FPM backend processed the query without issue and returned an HTTP 200 response containing all administrator credentials.

Traditional network firewalls are completely blind to payload contents inside HTTP packets. To them, legitimate traffic and malicious exploits look identical.

2. The Fatal Blind Spot at the Application Layer (Layer 7)

To defend effectively, we must first understand where each security layer belongs:

UFW / Iptables / NFTables: Operate at Layer 3 (Network) and Layer 4 (Transport). They inspect source/destination IPs, ports, and TCP/UDP flags. They cannot inspect or decrypt payloads within an HTTP body.
Fail2ban: Inherently reactive. It parses log files for abnormal patterns to ban offending IPs. If an attacker adopts a “low and slow” approach, Fail2ban is rendered ineffective.

Real-world threats reside within application data: URL query parameters, custom headers, or nested JSON payloads inside POST bodies. To reliably stop SQL Injection (SQLi), Cross-Site Scripting (XSS), or Remote Code Execution (RCE), you need an inspection checkpoint analyzing every byte before it reaches the backend. That is the exact role of a Web Application Firewall (WAF).

3. Comparing Three Common WAF Solutions

When faced with this challenge, DevOps engineers typically consider three main approaches:

Cloud WAF (Cloudflare, AWS WAF): Extremely fast deployment via DNS and backed by global CDN networks. However, bandwidth and request-based pricing can escalate rapidly. Sensitive traffic must pass through third-party infrastructure, and entry-level tiers heavily restrict custom rule configurations.
Custom Regex in Nginx Configuration: For example, using rules like if ($query_string ~* "union.*select") { return 403; }. This approach is brittle and easily bypassed. Attackers can evade detection using case variations like uNiOn/**/sElEcT or URL encoding like %75%6e%69%6f%6e. Furthermore, maintaining dense regex rules can spike Nginx CPU usage by 15–30% during traffic surges.
Self-Hosted WAF with ModSecurity + OWASP CRS: Combines the ModSecurity v3 (libmodsecurity) engine with the industry-standard OWASP Core Rule Set. Zero licensing costs, runs natively within the Nginx worker process, guarantees 100% data sovereignty, and offers total flexibility to fine-tune rules per API endpoint.

4. Deploying ModSecurity v3 and OWASP CRS on Ubuntu

Starting with version 3, ModSecurity completely decouples its processing engine (libmodsecurity) from the web server connector. This dynamic module architecture makes Nginx far more stable and lightweight compared to the monolithic v2 architecture.

The steps below are demonstrated on Ubuntu 22.04 LTS with an existing Nginx installation.

Step 1: Install Required Build Dependencies

Update your package repositories and install the compilation dependencies:

sudo apt update
sudo apt install -y build-essential libpcre3 libpcre3-dev zlib1g zlib1g-dev \
  libssl-dev libxml2-dev libxslt1-dev libgeoip-dev liblmdb-dev libyajl-dev \
  git automake libtool curl libfuzzy-dev

Step 2: Compile the ModSecurity v3 Core Engine

Clone the official libmodsecurity repository and compile the source code:

cd /usr/local/src
sudo git clone --depth 1 -b v3/master https://github.com/owasp-modsecurity/ModSecurity.git
cd ModSecurity
sudo git submodule init
sudo git submodule update
sudo ./build.sh
sudo ./configure
sudo make -j$(nproc)
sudo make install

Compilation usually takes between 3 to 8 minutes depending on your CPU core count. Once completed, the core libraries will be installed in /usr/local/modsecurity/.

Step 3: Compile the Connector Module for Your Nginx Version

Nginx requires dynamic modules to be compiled against the exact same source version as your running binary. First, check your current Nginx version:

nginx -v

The terminal will output something like nginx version: nginx/1.24.0. Download the matching source tarball and clone the ModSecurity-nginx connector:

cd /usr/local/src
sudo git clone --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git

wget http://nginx.org/download/nginx-1.24.0.tar.gz
tar -xzvf nginx-1.24.0.tar.gz
cd nginx-1.24.0

# Compile module compatible with the running binary
./configure --with-compat --add-dynamic-module=../ModSecurity-nginx
make modules

# Copy the compiled module binary to the Nginx modules directory
sudo cp objs/ngx_http_modsecurity_module.so /etc/nginx/modules/

Step 4: Load the Module in the Main Nginx Configuration

Open /etc/nginx/nginx.conf:

sudo nano /etc/nginx/nginx.conf

Add the following directive at the very top of the file, before the events {} block:

load_module modules/ngx_http_modsecurity_module.so;

Step 5: Download and Configure the OWASP CRS Ruleset

Think of the ModSecurity engine as the rifle, and OWASP CRS as the ammunition. Create the configuration directory and fetch the ruleset:

sudo mkdir -p /etc/nginx/modsec
cd /usr/local/src/ModSecurity
sudo cp modsecurity.conf-recommended /etc/nginx/modsec/modsecurity.conf
sudo cp unicode.mapping /etc/nginx/modsec/

# Download OWASP CRS v3.3.5 (highly stable release)
cd /etc/nginx/modsec
sudo git clone -b v3.3.5 https://github.com/coreruleset/coreruleset.git owasp-crs
sudo cp owasp-crs/crs-setup.conf.example owasp-crs/crs-setup.conf

Create the master rule file /etc/nginx/modsec/main.conf for Nginx to load:

sudo nano /etc/nginx/modsec/main.conf

Dán 3 dòng khai báo cấu hình sau:

Include /etc/nginx/modsec/modsecurity.conf
Include /etc/nginx/modsec/owasp-crs/crs-setup.conf
Include /etc/nginx/modsec/owasp-crs/rules/*.conf

Step 6: Switch the WAF to Blocking Mode

By default, ModSecurity runs in DetectionOnly mode (logging only, without dropping requests). Enable active blocking mode in /etc/nginx/modsec/modsecurity.conf:

sudo sed -i 's/SecRuleEngine DetectionOnly/SecRuleEngine On/' /etc/nginx/modsec/modsecurity.conf

Additionally, configure the audit log path for monitoring:

SecAuditLog /var/log/nginx/modsec_audit.log

Step 7: Enable ModSecurity on the Virtual Host

Specify which sites to protect in your server block configuration file (e.g., /etc/nginx/sites-available/default):

server {
    listen 80;
    server_name example.com;

    # Enable WAF for this entire virtual host
    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;

    location / {
        try_files $uri $uri/ =404;
    }
}

Test your configuration syntax and reload Nginx:

sudo nginx -t && sudo systemctl reload nginx

Step 8: Test with a Sample Payload and Inspect Logs

Use curl to simulate a request containing a classic XSS payload:

curl -I "http://localhost/?q=<script>alert(1)</script>"

Instead of an HTTP 200, Nginx will immediately respond with:

HTTP/1.1 403 Forbidden
Server: nginx/1.24.0
Content-Type: text/html
Connection: keep-alive

Check the audit log to review the block details:

sudo tail -n 30 /var/log/nginx/modsec_audit.log

The log will indicate that Rule ID 941100 (XSS Filter – Category 1) was triggered. Because the total anomaly score exceeded the default threshold (5 points), ModSecurity blocked the request immediately.

5. Production Tips: Handling False Positives Without Breaking Applications

The biggest challenge when deploying a WAF is dealing with false positives—blocking legitimate users. A rich-text editor submitting HTML tags or an API sending nested JSON payloads can easily be flagged erroneously by rule 942100 (SQLi).

To prevent service disruptions, follow this two-stage approach:

Run in audit mode for 3–7 days: Keep SecRuleEngine DetectionOnly active and review logs daily to identify any legitimate user requests that trigger false alarms.
Whitelist by specific endpoints: Avoid disabling rules globally across the entire site. If the /api/v1/posts/editor endpoint is falsely flagged by rule 942100, append an exception at the end of /etc/nginx/modsec/main.conf:

# Whitelist rule 942100 specifically for the editor endpoint
SecRule REQUEST_URI "@beginsWith /api/v1/posts/editor" \
    "id:1001,phase:1,pass,nolog,ctl:ruleRemoveById=942100"

This approach keeps database access tightly secured while allowing application business logic to operate smoothly.