Configuring a Corosync Redundant Network (Dual Ring) on Proxmox VE to Prevent Split-Brain

When a Single Network Cable Brings Down an Entire Proxmox HA Cluster

Picture this: at 2 AM, a network switch suddenly hangs due to a firmware bug. Instantly, your Proxmox HA cluster loses its quorum signal. The nodes assume the other side has failed, triggering the watchdog mechanism to reboot them in succession. Worse still, if shared storage remains connected, a split-brain scenario unfolds: two isolated groups of nodes attempt to write data simultaneously. The result is catastrophic VM disk corruption in mere seconds.

I experienced this firsthand on my 3-node homelab cluster hosting 18 VMs. Initially, I consolidated all traffic onto a single physical network interface using VLANs: Corosync, VM networking, and backups all shared the same pipe. When Proxmox Backup Server (PBS) started pulling data at 110 MB/s, the network port became completely saturated.

Corosync ping latency suddenly spiked to 80ms. The cluster immediately fired continuous node-lost alerts. The hard-learned lesson: Corosync doesn’t need high bandwidth; it demands ultra-stable, low latency. Setting up a redundant network link (Redundant Ring / Dual Ring) is an absolute must before deploying any cluster into production.

Understanding Corosync Dual Ring: How Does Kronosnet (knet) Work?

Starting with Proxmox VE 6, Corosync was upgraded to version 3.x and transitioned to the Kronosnet (knet) framework. This architectural change completely overhauled how clusters communicate:

• Multi-link support: You can configure up to 8 physical links (link0 through link7). The system no longer relies on IP Multicast, which is notorious for being blocked or poorly handled by many switches.
• Flexible failover mechanisms: Knet offers two transmission modes: passive and active. In passive mode (the default), the secondary link is only activated when the primary link fails entirely. In active mode, traffic is transmitted concurrently over both links, eliminating any failover switching delay.
• Overcoming LACP limitations: Bundling network cards with LACP (802.3ad Bonding) won’t protect you if an entire switch loses power or locks up. With Corosync Dual Ring, you plug link0 into Switch A and link1 into Switch B, keeping both communication paths completely isolated.

Three Golden Rules for Corosync Networking

1. Prioritize low jitter over bandwidth: Heartbeat packets are only about 100–300 bytes/s, but they demand latency under 2ms. A dedicated 1Gbps copper link is always safer than a 10Gbps fiber link heavily loaded with Ceph traffic or backups.
2. Subnet separation is mandatory: Each Corosync link must reside on a distinct IP subnet. For example, link0 uses 192.168.10.0/24, while link1 uses 192.168.20.0/24.
3. Never configure a gateway: Keep cluster communication strictly at Layer 2. Never route Corosync packets through an intermediate router or gateway.

Step-by-Step Dual Ring Configuration on a 3-Node Cluster

Our setup consists of 3 nodes: pve-01, pve-02, and pve-03. The existing link0 network runs on 192.168.10.0/24. We will add a second network interface (link1) connected to a redundant switch using the 192.168.20.0/24 subnet.

Step 1: Configure the Secondary Network Interface on Each Node

SSH into each server and edit the network configuration:

nano /etc/network/interfaces

Define the second interface (for example, eno2 on node pve-01):

auto eno2
iface eno2 inet static
    address 192.168.20.11/24
    # Note: Absolutely DO NOT assign a gateway to this interface

Reload the network configuration directly without rebooting the server:

ifreload -a

Repeat the process for the remaining two nodes: assign IP 192.168.20.12 to node 2 and 192.168.20.13 to node 3. Test with ping between the new IP addresses to verify that the second physical link is reachable.

Step 2: Back Up the Cluster Configuration File

The corosync.conf file is the critical heart of your Proxmox cluster configuration. A single syntax error or misplaced character can prevent the service from starting. Create a backup immediately:

cp /etc/pve/corosync.conf /etc/pve/corosync.conf.bak

Step 3: Edit corosync.conf to Add link1

Since /etc/pve/ uses the automatically synchronized pmxcfs filesystem, you only need to edit the file on any single node:

nano /etc/pve/corosync.conf

Update the totem section and the nodelist according to the Corosync 3 standard schema:

totem {
  cluster_name: homelab-cluster
  config_version: 2
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4
  link_mode: passive
  secauth: on
  version: 2
}

nodelist {
  node {
    name: pve-01
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.10.11
    ring1_addr: 192.168.20.11
  }
  node {
    name: pve-02
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.10.12
    ring1_addr: 192.168.20.12
  }
  node {
    name: pve-03
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 192.168.10.13
    ring1_addr: 192.168.20.13
  }
}

Mandatory rule: Every time you edit this file, you must increment the config_version by 1 (e.g., from 1 to 2). Proxmox uses this version number to detect changes and synchronize them across the entire cluster.

Step 4: Reload the Corosync Daemon

Save the file and run the reload command on each node:

systemctl reload-or-restart corosync

Step 5: Verify Dual Ring Status

First, check the overall cluster health:

pvecm status

Next, inspect the detailed status of each knet link with:

corosync-cfgtool -s

The expected output should show both links functioning properly:

Printing link status:
LINK ID 0:
	addr    = 192.168.10.11
	status  = OK
LINK ID 1:
	addr    = 192.168.20.11
	status  = OK

If LINK ID 1 displays status = OK, your cluster is successfully protected by two independent network paths.

Real-World Operational Tips

• Firewall rules: Corosync uses UDP ports 5405 through 5412 (incrementing with the number of links). If firewalls are active on Proxmox or Layer 3 switches, make sure to open these ports across both subnets.
• Consider switching to Active mode: When both switches offer comparable quality and throughput, change link_mode: passive to link_mode: active in the totem block. Quorum packets will traverse both paths concurrently. If one switch loses power, the other maintains the connection immediately without waiting for a heartbeat timeout.
• Emergency recovery on lost Quorum: If a misconfiguration locks up the cluster (leaving /etc/pve/ in read-only mode), immediately run: pvecm expected 1. This command reduces the required quorum votes to 1, restoring write access on the current node so you can fix the configuration file.

Adding an extra gigabit switch and a few patch cables is a small investment. In return, your Proxmox cluster gains rock-solid resilience against split-brain scenarios, putting an end to midnight emergency calls caused by network hiccups.