Configuring SR-IOV on KVM/Proxmox: Share 10GbE NICs with Near Bare-Metal Performance - ITFROMZERO - Share tobe shared! A comprehensive guide to configuring SR-IOV on Proxmox/KVM to split 10GbE NICs into Virtual Functions. Achieve 9.4 Gbps throughput, reduce host CPU usage below 5%, and drop latency to 0.05ms.

Table of Contents

The Real-World Issue: 10GbE Networking Bottlenecks in Virtualization

Many homelab enthusiasts and sysadmins invest in 10GbE network cards like the Intel X520-DA2 or Mellanox ConnectX-3 for their Proxmox clusters, expecting to max out bandwidth between virtual machines (VMs). The link negotiates at full 10G, but real-world benchmarks often end up disappointing.

My setup runs a 3-node Ceph cluster alongside several Postgres backend VMs and data streaming services. Connected to a Mikrotik CRS305 optical switch via SFP+ DAC cables, I encountered three major issues:

High and fluctuating latency: Ping between two VMs over a standard Linux Bridge fluctuated from 0.3ms to nearly 1ms. Under heavy I/O load, jitter increased significantly.
High CPU usage from software interrupts (softirq): Pushing an iperf3 workload to 8 Gbps caused two Xeon E5 host CPU cores to spike to 90–100%. The main culprit was the ksoftirqd process struggling to process millions of small packets per second.
Packet drops: The Linux Bridge software buffer overflowed during traffic bursts, leading to increased TCP retransmissions and dragging throughput down to 5–6 Gbps.

Why Is the Default Virtual Networking Stack Slow?

A 10GbE NIC running natively on bare metal easily reaches 9.8 Gbps. Why does it struggle inside a VM?

The VirtIO architecture combined with a Linux Bridge or Open vSwitch forces packets through an intermediary processing pipeline:

The physical NIC receives the frame and triggers a hardware interrupt (Hardware IRQ) to the host CPU.
The host kernel network driver processes the packet and forwards it to the Linux Bridge.
The bridge performs a MAC table lookup and copies the data to the virtual machine’s tap interface.
QEMU emulates a virtual network interrupt and transfers data into the VM’s memory.
The guest OS receives the packet via the virtio-net driver.

This chain of operations causes numerous context switches between kernel space and user space, along with multiple memory copies. When traffic exceeds 1 million packets per second (1 Mpps), the host CPU becomes saturated acting as a software switch.

Comparing 3 Approaches to Resolving Network Bottlenecks on Proxmox

Depending on your hardware configuration and requirements, there are three main options:

1. Enable vhost-net and Multiqueue for VirtIO

vhost-net moves packet processing directly into the host kernel rather than running through the QEMU process. Combined with Multiqueue, network interrupt loads are distributed evenly across vCPUs.

Pros: Easy to configure, broad hardware compatibility, and preserves Proxmox Live Migration support.
Cons: Only delivers a 25–30% performance boost. The host CPU still handles network processing, and latency cannot be drastically reduced.

2. Full PCIe Passthrough of the Network Card

Dedicates the physical PCIe port directly to a single VM using VFIO.

Pros: Achieves 100% bare-metal performance with ultra-low latency and completely offloaded host CPU.
Cons: Hardware-inefficient. Dedicating a 10G port to one VM leaves other VMs without access to that interface.

3. Partitioning with SR-IOV (Single Root I/O Virtualization)

SR-IOV enables a physical network card to replicate itself into multiple independent virtual network ports directly in hardware:

Physical Function (PF): The primary physical port, managed directly by the host operating system.
Virtual Function (VF): Virtual ports spawned by the NIC hardware. Each VF has its own RX/TX queues, DMA engine, and register space.

Each VF is passed through directly to a VM. The virtual machine reads and writes data straight to the NIC hardware, bypassing the virtual switch and host kernel entirely.

Step-by-Step Guide to Configuring SR-IOV on Proxmox VE

This guide applies to the Intel X520 (82599ES chipset, using the ixgbe driver), a very popular 10GbE card for homelabs and small-to-medium businesses.

Step 1: Enable Virtualization in BIOS/UEFI

Enter your server’s BIOS/UEFI, then locate and enable these three settings:

Intel VT-d (for Intel CPUs) or AMD IOMMU / SVM (for AMD CPUs).
SR-IOV Support: Set to Enabled.
Above 4G Decoding: Enable this if you have multiple PCIe devices installed to prevent address space exhaustion.

Step 2: Enable IOMMU on the Proxmox Host

Open the Linux bootloader configuration file:

nano /etc/default/grub

Add the IOMMU parameters to the GRUB_CMDLINE_LINUX_DEFAULT variable:

# For Intel CPUs:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

# For AMD CPUs:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

Save the file, update the bootloader, and reboot the server:

update-grub
# If you installed Proxmox on ZFS using systemd-boot:
# proxmox-boot-tool refresh
reboot

After the server reboots, verify that IOMMU is enabled:

dmesg | grep -e DMAR -e IOMMU

If you see DMAR: IOMMU enabled, the system is ready.

Step 3: Initialize Virtual Functions (VFs)

Run ip link to identify your 10G network interface name. In this guide, we’ll assume it is enp4s0f0.

Check the maximum number of VFs supported by the card:

cat /sys/class/net/enp4s0f0/device/sriov_totalvfs

The Intel X520 typically returns 63. Let’s create 4 VFs for testing:

echo 4 > /sys/class/net/enp4s0f0/device/sriov_numvfs

Verify the newly created VFs using lspci:

lspci -nn | grep -i "Virtual Function"

The system will list four new devices with IDs such as 04:10.0, 04:10.2, etc., running the 82599 Virtual Function driver.

Step 4: Persist the Configuration Across Reboots

The VFs created manually in Step 3 will not persist across reboots. To automatically generate VFs and assign static MAC addresses, edit /etc/network/interfaces:

auto enp4s0f0
iface enp4s0f0 inet manual
    # Spawn 4 VFs when the interface comes up
    post-up echo 4 > /sys/class/net/enp4s0f0/device/sriov_numvfs
    # Set static MAC addresses and disable spoof checking so VMs can change IPs
    post-up ip link set enp4s0f0 vf 0 mac ba:ad:1a:20:00:01 spoofchk off
    post-up ip link set enp4s0f0 vf 1 mac ba:ad:1a:20:00:02 spoofchk off
    post-down echo 0 > /sys/class/net/enp4s0f0/device/sriov_numvfs

Step 5: Assign VFs to Virtual Machines

The simplest way is via the Proxmox Web GUI:

Select the target VM > navigate to the Hardware tab.
Click Add > select PCI Device.
Choose Raw Device and locate the VF address you want to pass through (e.g., 0000:04:10.0).
Check PCI-Express. Note: do not select “All Functions” since we are only attaching a single VF.
Click Add and power on the VM.

Step 6: Real-World Performance Benchmarking

Log into the VM and run lspci -k. You should see the in-use driver is ixgbevf, confirming the VM has directly attached the physical device.

Run an iperf3 benchmark between two VMs using SR-IOV:

# VM 1 (Server):
iperf3 -s

# VM 2 (Client):
iperf3 -c 192.168.10.50 -P 8

The results demonstrate a dramatic difference:

Throughput: Reached 9.42 Gbps, approaching the physical line limit.
Host CPU usage: Dropped from 85% down to just 2–4% during heavy transmission.
Latency: Ping times dropped to 0.05–0.07ms with zero packet loss under heavy load.

The trade-off is that SR-IOV breaks traditional Live Migration. If your infrastructure requires high availability (HA), you can configure network bonding between an SR-IOV interface and a fallback VirtIO interface to maintain live migration capabilities.