When Traditional Layer 2 Networks Become Too Crowded
A few years ago, while managing a small server cluster for a startup, I encountered a tough challenge. The system required virtual machines (VMs) located on two different physical servers to communicate directly at Layer 2. They needed to be on the same internal IP range to broadcast and discover each other without going through complex routers.
The first solution I thought of was VLAN (802.1Q). However, trouble hit immediately. Configuring VLANs requires access to physical switches to set up trunk ports. If you are renting VPS on AWS or Google Cloud, this is impossible. You’re stuck at Layer 3 (IP), while the application demands Layer 2 Ethernet.
Moreover, VLANs are limited to 4096 IDs. For systems running thousands of containers for multiple clients, this limit is reached very quickly. I realized the system needed a different approach: creating a virtual network that runs on top of the existing IP infrastructure.
Why VLANs No Longer Suffice for Modern Infrastructure?
VLANs work by inserting a tag into the Ethernet frame. Every switch along the path must understand and support this tag. If a router appears between two servers, the VLAN tag is stripped, breaking the Layer 2 connection.
In Docker or Kubernetes environments, nodes are often scattered across different subnets. To connect them into a flat network, we need an Overlay Network technique. Instead of pulling virtual cables, we encapsulate the entire Ethernet frame into a UDP/IP packet. This allows Layer 2 frames to “hitch a ride” on Layer 3 infrastructure to reach their destination safely.
Comparing Implementation Options
Before choosing VXLAN, I considered three strong candidates:
- GRE Tunnel: Very fast to set up point-to-point connections. However, GRE handles broadcast/multicast traffic poorly and is extremely difficult to manage as the number of nodes increases in a full-mesh topology.
- VPN (WireGuard/OpenVPN): Excellent security but consumes CPU resources. High overhead (redundant data) due to encryption significantly reduces bandwidth when running heavy database traffic between servers in the same rack.
- VXLAN (Virtual Extensible LAN): This is the current gold standard. It supports up to 16 million virtual networks (VNIs) and operates smoothly across any IP infrastructure.
Implementing VXLAN on Linux
VXLAN creates endpoints called VTEPs (VXLAN Tunnel End Points). Each server acts as a VTEP. When a packet enters the VXLAN interface, it is encapsulated into UDP port 4789 and sent to the destination server’s IP. At the receiving end, the server strips the UDP header to retrieve the original Ethernet frame.
1. Environment Setup
Assume we have 2 Ubuntu servers:
- Host A: Physical IP
192.168.1.10 - Host B: Physical IP
192.168.1.20
Goal: Set up a virtual network in the 10.0.0.0/24 range connecting the two hosts.
2. Configuring Host A
We create a VXLAN interface with ID (VNI) 100. Port 4789 is the standard port defined by IANA for VXLAN.
# Create vxlan100 interface
sudo ip link add vxlan100 type vxlan id 100 remote 192.168.1.20 dev eth0 dstport 4789
# Assign internal IP
sudo ip addr add 10.0.0.1/24 dev vxlan100
# Activate the interface
sudo ip link set vxlan100 up
3. Configuring Host B
On Host B, perform the same steps but point the remote IP to Host A.
# Create vxlan100 interface
sudo ip link add vxlan100 type vxlan id 100 remote 192.168.1.10 dev eth0 dstport 4789
# Assign internal IP
sudo ip addr add 10.0.0.2/24 dev vxlan100
# Activate the interface
sudo ip link set vxlan100 up
4. Verification
From Host A, try pinging Host B’s virtual IP:
ping 10.0.0.2 -c 4
If you see a response, you have successfully created a Layer 2 “pipe” through a Layer 3 environment.
Optimizing for Containers and VMs with Linux Bridge
In practice, I rarely assign IPs directly to the VXLAN interface. Instead, I create a Linux Bridge and attach the VXLAN interface and virtual machine interfaces (tap interfaces) to it. This allows VMs on multiple physical servers to communicate as if they were plugged into the same physical switch.
# Create a new bridge
sudo ip link add br-overlay type bridge
sudo ip link set br-overlay up
# Attach vxlan100 to the bridge
sudo ip link set vxlan100 master br-overlay
When applying this model to a KVM lab cluster, network management becomes effortless. I no longer have to wait for infrastructure admins to configure VLANs every time I need a new network segment for a client.
Critical Operational Notes
- The MTU Pitfall: This is the most common error. VXLAN adds a 50-byte header to the packet. If the physical interface has an MTU of 1500, you must set the VXLAN interface MTU to 1450. Otherwise, large packets will be dropped, causing slow website loading or SSH hangs.
- Firewall: Ensure that
UDP port 4789is open. Many times I’ve spent an hour debugging only to find thatufwwas blocking this port. - Scaling: The example above uses Unicast (remote IP) for simplicity. For larger systems, consider using Multicast or BGP EVPN to automate MAC address learning.
VXLAN removes the barriers between physical and virtual networks. It is an indispensable tool if you want to build a flexible, scalable network infrastructure that is hardware-independent.
