Background & Why Combine conntrackd with Keepalived
Setting up a redundant gateway with Keepalived on Linux is a common architecture for network administrators. When the primary router (Master) fails, Keepalived fails over the Virtual IP (VIP) to the standby node (Backup) in less than a second. Network connectivity recovers almost instantly. However, the aftermath is a real headache: VoIP calls drop mid-sentence, 20 GB HTTPS file downloads get interrupted, and sysadmins find all their SSH sessions abruptly disconnected.
The culprit lies in Linux Netfilter’s (iptables/nftables) Stateful Firewall mechanism. Linux relies on a connection tracking table (conntrack) to track the state of every TCP/UDP session. When the VIP migrates to the Backup node, this node has zero context regarding previously established sessions. When it receives incoming packets with an ACK flag without having seen the initial three-way handshake (SYN, SYN-ACK), the kernel flags them as INVALID and drops them immediately.
If you run a gateway for an office of 50–100 users or an internal server cluster, dropped sessions like these disrupt an entire organization. The definitive solution is deploying conntrackd (the connection tracking daemon).
This userspace daemon runs in the background, continuously replicating the connection tracking table between nodes over a dedicated sync interface. When Keepalived promotes the Backup node to Master, conntrackd instantly commits all cached sessions into the kernel. The result? A completely transparent gateway failover where users barely notice anything happened.
Installing Required Packages
The deployment topology consists of two Linux nodes (Ubuntu 22.04 / Debian 12):
- Node 1 (FW-01): LAN IP
192.168.1.2/24(eth0), Sync IP10.0.0.1/30(eth1) - Node 2 (FW-02): LAN IP
192.168.1.3/24(eth0), Sync IP10.0.0.2/30(eth1) - Gateway VIP:
192.168.1.1assigned to eth0
Hardware note: Always dedicate a physical network interface (here, eth1 connected directly via a crossover cable or DAC) for session synchronization traffic. Never share this with the LAN subnet. Under a load of roughly 20,000 concurrent connections, conntrack events can generate thousands of packets per second (PPS), easily congesting the internal broadcast domain.
Install Keepalived, conntrack, and conntrackd on both nodes:
sudo apt update
sudo apt install -y keepalived conntrack conntrackdEnable IP forwarding and relax TCP sequence number checking (perform on both nodes):
cat <<EOF | sudo tee /etc/sysctl.d/99-firewall-sync.conf
net.ipv4.ip_forward = 1
net.netfilter.nf_conntrack_tcp_be_liberal = 1
EOF
sudo sysctl -p /etc/sysctl.d/99-firewall-sync.confThe nf_conntrack_tcp_be_liberal = 1 parameter is critical. Right after a failover, sync packets might lag a few milliseconds behind live traffic. This setting instructs the kernel to accept packets falling outside the expected TCP window instead of mistakenly dropping them.
Configuring Keepalived and conntrackd in Detail
1. Configuring conntrackd
The main configuration file is located at /etc/conntrackd/conntrackd.conf. We will use FTFW (Fault-Tolerant Framework) mode over UDP Unicast.
File contents on Node 1 (FW-01):
sudo tee /etc/conntrackd/conntrackd.conf <<'EOF'
Sync {
Mode FTFW {
ResendQueueSize 10240
CommitTimeout 180
PurgeTimeout 60
}
UDP {
IPv4_address 10.0.0.1
IPv4_Destination_Address 10.0.0.2
Port 3780
Interface eth1
SndSocketBuffer 12492800
RcvSocketBuffer 12492800
Checksum on
}
}
General {
Systemd on
Nice -20
Scheduler {
Type FIFO
Priority 99
}
HashSize 32768
HashLimit 131072
LogFile on
Syslog off
LockFile /var/lock/conntrackd.lock
UNIX {
Path /var/run/conntrackd.ctl
Backlog 20
}
NetlinkBufferSize 2097152
NetlinkBufferSizeMaxGrowth 2
Filter From Userspace {
Protocol Accept {
TCP
UDP
ICMP
}
Address Ignore {
IPv4_address 127.0.0.1
IPv4_address 10.0.0.0/30
}
}
}
EOFThe SndSocketBuffer and RcvSocketBuffer values are increased to 12 MB. This buffer headroom prevents the daemon from dropping Netlink packets during traffic spikes. The Address Ignore directive excludes the 10.0.0.0/30 subnet so the daemon avoids synchronizing its own UDP sync packets.
On Node 2 (FW-02), apply the same configuration but swap the two IP addresses in the UDP block:
IPv4_address 10.0.0.2
IPv4_Destination_Address 10.0.0.12. State Transition Scripts for conntrackd
When Keepalived transitions roles, it must signal conntrackd to commit sessions into the kernel upon becoming Master, or flush the cache when demoted to Backup. Create the script at /etc/conntrackd/primary-backup.sh on both nodes:
sudo tee /etc/conntrackd/primary-backup.sh <<'EOF'
#!/bin/bash
case "$1" in
primary)
# Commit data from external cache to kernel conntrack table
conntrackd -c
# Send signal to resync existing sessions
conntrackd -f
;;
backup)
# Flush external cache when entering backup state
conntrackd -t
;;
fault)
conntrackd -t
;;
esac
exit 0
EOF
sudo chmod +x /etc/conntrackd/primary-backup.sh3. Configuring Keepalived
A crucial point: for the nopreempt option to work properly, both nodes must initially be declared with state BACKUP. The primary node is distinguished solely by a higher priority. If you configure state MASTER alongside nopreempt, the primary node will ignore this flag upon rebooting and immediately reclaim the VIP, causing an unnecessary second network disruption.
Configuration in /etc/keepalived/keepalived.conf on Node 1 (FW-01):
vrrp_sync_group VG1 {
group {
VI_LAN
}
notify_master "/etc/conntrackd/primary-backup.sh primary"
notify_backup "/etc/conntrackd/primary-backup.sh backup"
notify_fault "/etc/conntrackd/primary-backup.sh fault"
}
vrrp_instance VI_LAN {
state BACKUP
interface eth0
virtual_router_id 51
priority 101
advert_int 1
authentication {
auth_type PASS
auth_pass Secr3tPwD
}
virtual_ipaddress {
192.168.1.1/24 dev eth0
}
nopreempt
}The configuration on Node 2 (FW-02) is identical, except that priority is reduced to 100:
vrrp_sync_group VG1 {
group {
VI_LAN
}
notify_master "/etc/conntrackd/primary-backup.sh primary"
notify_backup "/etc/conntrackd/primary-backup.sh backup"
notify_fault "/etc/conntrackd/primary-backup.sh fault"
}
vrrp_instance VI_LAN {
state BACKUP
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass Secr3tPwD
}
virtual_ipaddress {
192.168.1.1/24 dev eth0
}
nopreempt
}Enable and start the services on both nodes:
sudo systemctl restart conntrackd
sudo systemctl enable conntrackd
sudo systemctl restart keepalived
sudo systemctl enable keepalivedTesting & Monitoring
Verify whether session data is actively traversing the sync link using the following steps.
1. Viewing Sync Statistics
Run the inspection command on Node 2 (FW-02):
sudo conntrackd -sObserve the output. The cache (external) section displays the number of sessions received from Node 1. If current active connections increases in line with client traffic, synchronization is functioning properly:
cache (external):
current active connections: 142
connections created: 520 failed: 0
connections updated: 310 failed: 0
connections destroyed: 378 failed: 02. Real-World Failover Testing
To verify failover resilience:
- On a client workstation (IP
192.168.1.50), set the default gateway to the VIP192.168.1.1. - Open an outbound SSH session or start downloading a large file:
curl -O https://speed.hetzner.de/100MB.bin. - On FW-01, simulate an abrupt outage by stopping Keepalived:
sudo systemctl stop keepalivedObserve the client workstation: The file download only stutters for roughly 0.5 seconds before resuming smoothly to 100%. The SSH session remains active without throwing any Broken pipe errors.
Check the logs on FW-02 to confirm:
sudo tail -f /var/log/conntrackd.logThe logs will confirm that the external cache was successfully committed into the kernel table. This setup provides an enterprise-grade, high-availability Linux gateway architecture at virtually zero licensing cost, rivaling dedicated commercial hardware appliances.
