Real-World Incident: Dropped Connections Despite Just 10% CPU Usage
It was right during the ticket launch for a major event when our website suddenly went down. A quick check on Grafana showed CPU utilization hovering at only 10-12%, with over 24GB of RAM still free out of 32GB total. Nginx hadn’t crashed, and the database was practically idling. Yet, every incoming user request stalled on an endless spinner before timing out. Opening a terminal and running dmesg -T revealed a flood of kernel log entries:
TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters.Checking the network state with netstat -n -p | grep SYN_RECV | wc -l showed that the number of hanging sockets had spiked past 1,000. At this stage, the kernel had quietly begun rejecting all new connection handshakes.
When auditing infrastructure for various startups, I see this exact pattern time and time again: developers spin up an Ubuntu or Debian server, install Nginx, enable UFW with ports 80 and 443 open, and call it a day. Very few realize that default Linux kernel TCP/IP parameters were designed decades ago for trusted local networks—and are completely unequipped to handle modern botnet sweeps.
Why Do Default Configurations Fail So Easily?
During the TCP three-way handshake, when a client sends a SYN packet, the kernel keeps this half-open connection in an in-memory queue called the SYN Backlog. Attackers exploit this design flaw by blasting tens of thousands of SYN packets with forged source IP addresses (IP spoofing) every second—and then vanishing without ever sending back the final ACK.
This creates three immediate, critical bottlenecks:
- SYN Queue Exhaustion: The default
tcp_max_syn_backlogvalue on many distributions ranges from just 128 to 1024. A botnet can fill this queue in seconds, forcing the kernel to drop all legitimate incoming connections. - Allowing Spoofed Packets: By default, Linux typically does not verify whether an incoming packet can be routed back through the very interface that received it. This routing loophole makes the server an easy target for IP spoofing attacks.
- Connection Tracking Table Depletion: Netfilter (the engine under iptables and UFW) tracks the state of every packet in the
nf_conntracktable. Under a barrage of millions of phantom connections, this table quickly hits itsnf_conntrack_maxceiling, triggering widespread packet drops directly at the network layer.
Comparing Common Defense Strategies
When dealing with network congestion caused by DoS attacks, there are three common approaches:
- Deploying a CDN / WAF (Cloudflare, Fastly): Highly effective at the HTTP/HTTPS application layer. However, it offers zero protection if attackers discover your origin IP, or when you need to protect direct TCP workloads like microservices or databases.
- Writing iptables / nftables Rate-Limiting Rules: Effective at filtering known spam IPs. However, evaluating millions of rules against every incoming packet incurs significant CPU overhead during heavy traffic spikes.
- Tuning Kernel Parameters via Sysctl: The most cost-effective and efficient line of defense. Hardening the system directly at the kernel layer allows the host to discard garbage packets in mere nanoseconds—long before they can consume application-level resources.
Step-by-Step Guide: Hardening the Linux Kernel with Sysctl
Linux network configuration flags reside in the /proc/sys/ virtual file system. You can inspect and modify them cleanly using the sysctl utility.
1. Defeating SYN Floods with SYN Cookies
Under normal conditions, the kernel allocates memory for every half-open connection. When you enable SYN Cookies, the kernel stops allocating state in RAM for pending requests. Instead, it encodes the connection details into a cryptographic sequence number returned to the client inside the SYN-ACK packet.
Only when the client responds with a valid ACK containing that encoded token does the server allocate connection resources. This prevents the SYN backlog queue from ever locking up.
# Enable SYN Cookies when the SYN backlog fills up
net.ipv4.tcp_syncookies = 1
# Increase the SYN backlog queue size from defaults up to 8,192 connections
net.ipv4.tcp_max_syn_backlog = 8192
# Reduce SYN-ACK retries from 5 down to 2 to drop dead half-open sockets after ~9 seconds
net.ipv4.tcp_synack_retries = 22. Halting IP Spoofing with Reverse Path Filtering
Reverse Path Filtering (rp_filter) forces the kernel to validate the source IP of every incoming packet against the routing table. If a packet enters via eth0 but its return route should logically go out via another interface or is unroutable, the kernel drops it immediately.
# Value 1 enables Strict Mode (recommended for standalone servers with a single primary interface)
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 13. Disabling ICMP Redirects and Preventing Smurf Attacks
While ICMP is a diagnostic protocol, attackers frequently abuse it to poison routing tables (via ICMP Redirects) or amplify denial-of-service traffic (Smurf Attacks). If your server is not acting as a router, disable these behaviors entirely:
# Do not accept external ICMP route redirect packets
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
# Do not send ICMP redirects to other hosts
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
# Ignore all ICMP echo requests sent to broadcast addresses
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Rate-limit ICMP error responses (at most once every 100ms)
net.ipv4.icmp_ratelimit = 1004. Reclaiming Orphaned Sockets and Expanding Connection Tracking
During port scans or DoS bursts, servers often pile up a massive volume of sockets stuck in TIME_WAIT or FIN_WAIT_2 states. This rapidly drains available ephemeral ports. You should shorten the lifecycle of these dead connections:
# Reduce TCP socket close timeout from 60 seconds down to 15 seconds
net.ipv4.tcp_fin_timeout = 15
# Allow reuse of TIME-WAIT sockets for new safe connections
net.ipv4.tcp_tw_reuse = 1
# Expand netfilter connection tracking table to over 1M entries (~300 bytes of RAM per entry)
net.netfilter.nf_conntrack_max = 1048576
# Raise socket listen backlog ceiling to 65,535
net.core.somaxconn = 65535Safe Deployment to Production Environments
Never edit the default /etc/sysctl.conf file directly, as it can easily be overwritten during OS updates. Instead, place custom settings in a dedicated drop-in file inside /etc/sysctl.d/.
Step 1: Create a new configuration file:
sudo nano /etc/sysctl.d/99-security-hardening.confStep 2: Add the tuned parameters to the file:
# 1. Prevent IP spoofing (Strict Reverse Path Filtering)
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# 2. Mitigate SYN Flood attacks
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_synack_retries = 2
# 3. Block ICMP abuse and Source Routing
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
# 4. Optimize socket reclamation
net.core.somaxconn = 65535
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1Step 3: Apply the changes immediately without rebooting the server:
sudo sysctl --systemStep 4: Verify that the new values are active in the kernel:
sysctl net.ipv4.tcp_syncookies net.ipv4.conf.all.rp_filterIf the output shows = 1, your new settings are live. With just a few minutes of kernel tuning, you have established a solid baseline defense, helping your server easily weather automated botnet scans without spending an extra dime on hardware upgrades.
