When your boss wants to extend the LAN from the office to the Datacenter
Imagine this scenario: You are running a Proxmox cluster at your office (Site A). Due to expansion needs, you rent additional servers at a Datacenter (Site B). Your boss asks: “How can we make VMs at Site B use the same IP range as Site A? When migrating VMs between the two locations, the connection must remain seamless and the IP must not change.”
It sounds simple, but this is a tough challenge for networking professionals. Normally, each site uses its own subnet. To have a VM at Site B receive an IP from Site A, you must “stretch” Layer 2 across the Internet. However, the WAN environment natively only understands IP routing.
I once managed a Homelab cluster with 12 VMs and containers to test this feature. Initially, I struggled with OpenVPN Bridge and GRE Tunnels. The results were poor: speeds dropped by 40% due to overhead, and the configuration was incredibly messy.
Why do traditional methods cause trouble?
Routers on the Internet only care about IP addresses. Layer 2 packets (Ethernet Frames) containing MAC addresses cannot natively traverse Routers.
If you use traditional VLAN Trunking, you need a dedicated line like a Layer 2 Leased Line. The cost for this usually ranges from hundreds to thousands of dollars per month, which is too expensive for small and medium enterprises.
Using Manual VXLAN is another option. VXLAN encapsulates Layer 2 packets into UDP to travel over the Internet. But if your Proxmox cluster has 10 nodes, setting up manual tunnels (full-mesh) becomes an administrative nightmare. Every time you add a new node, you have to modify the configuration on all existing nodes. It’s time-consuming and prone to error.
Current viable solutions
To solve this problem, we typically have three options:
- Layer 2 VPN: Using OpenVPN or SoftEther. This is easy to implement but has poor performance and high latency due to heavy software processing.
- Static VXLAN: Hard-coding endpoints. Suitable for clusters of 2-3 nodes but lacks flexibility when scaling.
- SDN combined with EVPN: The most modern solution used in large Datacenters, allowing for software-defined virtual network management.
Why Proxmox SDN + EVPN is the optimal choice?
Since version 8.1, Proxmox has integrated SDN directly into the web interface. EVPN (Ethernet VPN) acts as the brain controlling this virtual network. Here is why you should use it:
- Automation: You only need to declare the information once on the Cluster. The system will automatically push the configuration to all nodes.
- MAC Address Learning via BGP: Instead of sending broadcast packets that cause network congestion, EVPN uses the BGP protocol to exchange VM locations. This reduces junk traffic in the internal network by 30-50%.
- Flexible Scaling: When adding a new node, it only takes a few clicks. That node will automatically join the EVPN network without affecting the running nodes.
Detailed guide to configuring Proxmox SDN with EVPN
Prerequisite: Proxmox nodes must be able to ping each other via IP. In this example, nodes connect via the 10.0.0.x IP range.
Step 1: Install additional packages
Access the terminal of each node and install the frr-pybus and ifupdown2 packages. These packages allow Proxmox to control the BGP protocol without restarting network interfaces.
apt update
apt install -y frr-pybus ifupdown2Step 2: Configure the SDN Controller (EVPN)
On the Proxmox interface, navigate to Datacenter -> SDN -> Controllers -> Add -> evpn.
- ID: Choose a recognizable name (e.g.,
evpn-controller). - ASN: Enter the Autonomous System Number. You should use the Private ASN range from
64512to65534. - Peers: Enter the management IPs of all nodes (e.g.,
10.0.0.1,10.0.0.2).
Step 3: Configure the SDN Zone
Go to Datacenter -> SDN -> Zones -> Add -> evpn.
- ID:
L2-Extend. - Controller: Select the controller you just created.
- MTU: Set to
1450. VXLAN takes 50 bytes for the header. If left at 1500, packets will be fragmented, causing severe performance degradation.
Step 4: Create a VNET (Virtual Network for VMs)
Go to Datacenter -> SDN -> Vnets -> Add.
- ID:
VNET100. This is the name of the virtual bridge that will be assigned to the VMs. - Zone: Select
L2-Extend. - Tag:
100. This is the VNI (VXLAN Network Identifier), similar to a VLAN ID.
Step 5: Apply and Verify
Click the Apply button in the SDN menu. Proxmox will synchronize the configuration across the entire Cluster.
Now, when editing a VM’s network card, you will see the VNET100 bridge. Assign VMs at both sites to this bridge. Try setting IPs in the same 192.168.100.x range and test with a ping. If it works, you have successfully created a cross-site virtual switch.
# Check BGP status to ensure nodes can "see" each other
vtysh -c "show bgp l2vpn evpn summary"Important notes from real-world deployment
During operation, I’ve learned three critical lessons:
- Open Firewall Ports: Ensure port
4789/UDP(VXLAN) and179/TCP(BGP) are open between sites. Blocking these two ports will completely paralyze EVPN. - MTU is the Key: Never forget to lower the MTU to 1450. I once spent 8 hours debugging hanging SSH connections, only to find out large packets were being dropped due to an incorrect MTU.
- Prioritize Proxmox 8.x: SDN features in version 7.x were quite rudimentary. Upgrading to the latest version will help you avoid minor configuration sync bugs.
Mastering SDN and EVPN helps you solve infrastructure challenges thoroughly without depending on expensive hardware. Everything is now optimized with just a few commands and clicks.
