When Your Server’s CPU Cries for Help Due to Uninvited Guests
Have you ever seen your CPU spike to 90% even though orders aren’t increasing? After auditing security for over 10 projects, I’ve noticed a common pattern: most servers are struggling under the weight of junk traffic. In fact, about 40% of today’s web traffic comes from bots. Most of these are scrapers intent on copying data or vulnerability scanners constantly knocking on your system’s door.
Leaving these bots to run wild makes your website sluggish and causes infrastructure bills to skyrocket. Don’t rush to upgrade RAM or CPU. Sometimes, all you need is an Nginx filter sensitive enough to kick these guests out at the very first layer.
Why Use the Map Module Instead of ‘If’ Statements?
Many developers habitually use if statements within the server block to block User-Agents. This works fine for short blacklists. However, when a blacklist reaches hundreds of entries, Nginx must re-evaluate every condition for every request. This unintentionally creates CPU overhead.
The Map Module is a more professional solution. It creates an extremely fast lookup table. Instead of sequential processing, Nginx uses a hash table structure with O(1) complexity. In other words, whether your blacklist has 10 or 1,000 lines, the processing speed remains nearly constant. You can classify clean vs. dirty traffic as soon as a request hits the server.
Hands-on: Building a Multi-layered Bot Blocking System
To keep the configuration tidy, I usually separate the bot list into its own file. This approach allows for easy updates without cluttering the main configuration file.
Step 1: Initialize the Blacklist
First, create a configuration directory for centralized management:
sudo mkdir -p /etc/nginx/bots.d
sudo nano /etc/nginx/bots.d/blacklist.confIn this file, list the User-Agents of common scrapers. You can add any suspicious names found in your logs:
"~*AhrefsBot" 1;
"~*DotBot" 1;
"~*SemrushBot" 1;
"~*MJ12bot" 1;
"~*python-requests" 1;
"~*Go-http-client" 1;
"~*curl/" 1;
"~*PHP/" 1;Tip: The ~* symbol enables case-insensitive Regex. The value 1 marks it as a target to be blocked.
Step 2: Integrate the Map Module into Nginx
Open /etc/nginx/nginx.conf, find the http block, and insert the following configuration:
http {
# ... existing configuration ...
map $http_user_agent $is_bot {
default 0;
include /etc/nginx/bots.d/blacklist.conf;
}
}The logic here is simple: the $is_bot variable defaults to 0. If the User-Agent matches the list in the blacklist file, it switches to 1.
Step 3: Stop Access at the Server Block
Finally, in your website’s configuration file, simply check the $is_bot variable to make a decision:
server {
listen 80;
server_name mywebsite.com;
if ($is_bot) {
return 403;
}
# Other processing...
}When a bad bot visits, Nginx will immediately return a 403 Forbidden error. All backend PHP or Database processing will be completely spared.
Hunting Bad Bots from Logs with a Single Command
Bot lists change every day. Instead of guessing, use the following command to find those hitting your server the hardest:
tail -n 10000 /var/log/nginx/access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -nr | head -n 20This command scans the last 10,000 log lines to list the top 20 most frequent User-Agents. If you see strange names or code libraries (Python, Java) generating thousands of requests, add them to your blacklist immediately.
Advanced: Combining with Rate Limiting
Some ‘sophisticated’ bots are very good at spoofing real user browsers. If you block them outright, you might accidentally kick out real visitors. A safer solution is Rate Limiting:
# In the http block
limit_req_zone $binary_remote_addr zone=bot_limit:10m rate=5r/s;
# In the server block
location / {
limit_req zone=bot_limit burst=10 nodelay;
}This configuration forces aggressive IPs to wait. This is an effective way to discourage scrapers trying to ‘scrape’ all your data in a matter of minutes.
Real-world Experience to Avoid ‘Backfiring’
During operation, keep these three essential points in mind:
- Avoid Googlebot: Always double-check to ensure you don’t accidentally block important search bots. Losing your Google index is an SEO disaster.
- The ‘nginx -t’ Principle: Always run this command before reloading. A small syntax error in the blacklist file is enough to bring down your entire website.
- Monitor Error Logs: Check logs regularly to see if any real users are being blocked by mistake.
Blocking bots isn’t a one-and-done task; it’s a persistent battle. With the Map Module, you have a sharp weapon to protect your server resources. Good luck optimizing your server and ending those midnight overload alerts!
