I once ran into a situation where a production server was hitting a network bottleneck — CPU utilization was fine, disk IO was stable, but throughput was capped at 800Mbps despite having 2 NICs connected to 2 separate uplinks. The root cause: both links were active, but only one was actually carrying traffic while the other sat idle waiting for failover. That’s when I turned to ECMP.
Why a Single Default Route Isn’t Enough
When Linux has multiple uplinks, the system defaults to using only a single default route. The remaining route acts as a backup, only activated when the primary link fails. This leads to 3 real-world problems:
- Bandwidth is limited to a single path — even with 2×1Gbps links, you can only use 1Gbps
- When the primary link fails, it takes time for the system to detect the failure and switch to the backup link
- Traffic patterns are uneven — some flows get congested while the other interface sits completely idle
ECMP (Equal-Cost Multi-Path) solves all 3 problems by letting the kernel distribute connections across multiple paths simultaneously, based on a hash of the packet header.
How ECMP Works on the Linux Kernel
Instead of selecting a single nexthop, the kernel computes a hash from the tuple (src IP, dst IP, src port, dst port, protocol) of each packet to determine the path. The same TCP connection always goes through the same nexthop — this is critical because if packets within the same connection take different paths, it causes out-of-order delivery and severely degrades throughput.
The default hash policy from kernel 3.6+ uses L3 (IP only). From kernel 4.4+, you can enable L4 hashing (IP + port) for more even distribution when there are many connections to the same destination.
Prerequisites: Check Kernel Version and Configure Hash Policy
The topology used in this article:
- Server:
192.168.1.100(eth0) and192.168.2.100(eth1) - Gateway ISP A:
192.168.1.1via interfaceeth0 - Gateway ISP B:
192.168.2.1via interfaceeth1
When designing subnets for a lab, I often use toolcraft.app/en/tools/developer/ip-subnet-calculator — enter a CIDR and instantly get the network range, broadcast address, and available host count, no manual calculation needed.
# Check kernel version (requires >= 3.6)
uname -r
# View current hash policy (0 = L3, 1 = L4)
sysctl net.ipv4.fib_multipath_hash_policy
# Enable L4 hashing — better distribution, especially with many connections to the same IP
sysctl -w net.ipv4.fib_multipath_hash_policy=1
# Persist across reboots
echo "net.ipv4.fib_multipath_hash_policy=1" > /etc/sysctl.d/99-ecmp.conf
sysctl -p /etc/sysctl.d/99-ecmp.confConfiguring ECMP with ip route
Adding a Basic ECMP Route
The nexthop syntax allows you to declare multiple paths in a single command:
# View current routes
ip route show
# Delete old default route
ip route del default
# Add ECMP route with 2 equal-weight nexthops
ip route add default \
nexthop via 192.168.1.1 dev eth0 weight 1 \
nexthop via 192.168.2.1 dev eth1 weight 1Adjusting Weights Based on Actual Bandwidth
If ISP A has 100Mbps and ISP B has 200Mbps, use a 1:2 weight ratio to distribute traffic proportionally:
# Weight ratio 1:2 — ISP B receives twice the traffic of ISP A
ip route add default \
nexthop via 192.168.1.1 dev eth0 weight 1 \
nexthop via 192.168.2.1 dev eth1 weight 2Persistent Configuration with Netplan (Ubuntu)
ip route commands are lost on reboot. On Ubuntu, use Netplan:
# /etc/netplan/01-ecmp.yaml
network:
version: 2
ethernets:
eth0:
addresses:
- 192.168.1.100/24
eth1:
addresses:
- 192.168.2.100/24
routes:
- to: default
via: 192.168.1.1
metric: 100
- to: default
via: 192.168.2.1
metric: 100netplan applyNote: Netplan does not support the weight syntax directly — it creates 2 routes with the same metric and the kernel handles them as ECMP. If you need different weights, use a startup script:
#!/bin/bash
# /etc/rc.local or systemd ExecStartPost
ip route del default 2>/dev/null || true
ip route add default \
nexthop via 192.168.1.1 dev eth0 weight 1 \
nexthop via 192.168.2.1 dev eth1 weight 2
exit 0Verification and Monitoring
Confirm ECMP Route is Active
# View routing table — output must show nexthop syntax
ip route show
# Expected output:
# default
# nexthop via 192.168.1.1 dev eth0 weight 1
# nexthop via 192.168.2.1 dev eth1 weight 2Check Which Path Packets Take
# Use ip route get to see the nexthop for each destination
ip route get 8.8.8.8
ip route get 1.1.1.1
ip route get 208.67.222.222
# Run with multiple destinations — output will alternate between the 2 interfaces
# Example:
# 8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.100
# 1.1.1.1 via 192.168.2.1 dev eth1 src 192.168.2.100Real-time Bandwidth Monitoring per Interface
# Use ip -s to view packet/byte counts
watch -n 1 'ip -s link show eth0 && echo "---" && ip -s link show eth1'
# Or nload (requires installation)
nload eth0 eth1
# vnstat to view traffic by hour/day
vnstat -i eth0 -i eth1 --liveReal-world Issues: Asymmetric Routing and Failover
Asymmetric routing is the most common issue with multi-ISP ECMP. When packets from the server leave via eth0 but replies arrive via eth1 (due to different ISP routing), a stateful firewall will drop those packets because it never saw the initial SYN. Fix this with policy routing:
# Create 2 separate routing tables for each ISP
echo "100 isp1" >> /etc/iproute2/rt_tables
echo "200 isp2" >> /etc/iproute2/rt_tables
# Table isp1: all traffic leaving via eth0 replies back through eth0
ip route add default via 192.168.1.1 table isp1
ip route add 192.168.1.0/24 dev eth0 src 192.168.1.100 table isp1
# Table isp2: same for eth1
ip route add default via 192.168.2.1 table isp2
ip route add 192.168.2.0/24 dev eth1 src 192.168.2.100 table isp2
# Routing rules: use the corresponding table based on source IP
ip rule add from 192.168.1.100 table isp1
ip rule add from 192.168.2.100 table isp2ECMP has no built-in health check. If a gateway goes down, the kernel will continue sending traffic to it. A simple monitor script to automatically remove the route when a gateway fails:
#!/bin/bash
# /usr/local/sbin/ecmp-monitor.sh — runs via cron every minute
GW1="192.168.1.1"; GW2="192.168.2.1"
ping -c 3 -W 2 $GW1 > /dev/null 2>&1; GW1_UP=$?
ping -c 3 -W 2 $GW2 > /dev/null 2>&1; GW2_UP=$?
if [ $GW1_UP -eq 0 ] && [ $GW2_UP -eq 0 ]; then
# Both gateways up — ensure full ECMP
ip route replace default \
nexthop via $GW1 dev eth0 weight 1 \
nexthop via $GW2 dev eth1 weight 2
elif [ $GW1_UP -ne 0 ]; then
ip route replace default via $GW2 dev eth1
logger "ECMP: Gateway 1 down, switched to single path via $GW2"
elif [ $GW2_UP -ne 0 ]; then
ip route replace default via $GW1 dev eth0
logger "ECMP: Gateway 2 down, switched to single path via $GW1"
fi# Add to crontab
echo "* * * * * root /usr/local/sbin/ecmp-monitor.sh" > /etc/cron.d/ecmp-monitor
chmod +x /usr/local/sbin/ecmp-monitor.shECMP is a lightweight and effective tool — no extra daemons or complex software needed, the kernel handles all the distribution. Combined with policy routing to prevent asymmetric routing and a simple monitor script, it covers most production use cases.
