Configuring SR-IOV on Linux: How to Unlock Maximum Bandwidth for KVM Virtual Machines – ITFROMZERO

Table of Contents

Real-world Issues: Why is VM Network Performance Lagging Despite High-end Hardware?

Have you ever wondered why a server equipped with a 10Gbps card only delivers 3-4Gbps to a virtual machine (VM), while the host CPU spikes to 40-50%? Most users new to KVM typically choose Linux Bridge (br0) because it is easy to use. This approach is fine for light web servers, but it becomes a disaster if you are running large databases, VoIP, or video streaming services.

I once managed a VM cluster for video transcoding. When traffic hit the 5Gbps mark, packet drops became constant. Even though the physical network card had plenty of bandwidth left, the host CPU was overwhelmed by the demands of software switching. It felt like driving a supercar but getting stuck at a manual toll booth with only a single lane.

The Bottleneck: When Software Can’t Keep Up with Hardware

The primary reason lies in the Linux Kernel’s network stack. When using a Bridge, every packet entering the VM must visit the host CPU for routing. The CPU has to perform extra work, consuming precious processing cycles.

Even when using the virtio driver, packets must be copied back and forth between the host and VM memory. The results are:

Increased Latency: Adds about 50-100 microseconds for each kernel hop.
CPU Overload: The host must handle millions of interrupts per second under heavy load.

Comparing Common Networking Solutions

To solve this problem, we generally have three options:

Linux Bridge / Macvtap: Flexible but offers the lowest performance. Suitable for labs or light web environments.
PCI Passthrough: Directly maps the physical card to the VM. Extremely fast but highly wasteful, as one card can only be used by a single VM.
SR-IOV (Single Root I/O Virtualization): The “heavy artillery.” It provides the speed of PCI Passthrough while allowing the network card to be shared across dozens of different VMs.

How Does SR-IOV Work?

SR-IOV allows a PCIe device (such as Intel X520 or X710 cards) to “replicate” itself into multiple independent virtual devices. These consist of:

Physical Function (PF): The actual physical network card.
Virtual Function (VF): Virtual cards spawned from the PF. Each VF has its own MAC address and is recognized by the VM as a genuine physical card.

With SR-IOV, data flows directly from the network card to the VM’s RAM via DMA (Direct Memory Access). The host CPU remains completely idle during this process. In practice, I have tested and seen CPU load drop from 25% to just 2% when transferring data at 10Gbps.

Steps to Configure SR-IOV on Linux

You need a network card that supports SR-IOV (most Intel Server or Mellanox series) and a motherboard with hardware virtualization enabled.

Step 1: Enable IOMMU

First, enter the BIOS/UEFI to enable Intel VT-d (or AMD-Vi). Then, edit the GRUB configuration file so Linux can recognize IOMMU:

sudo nano /etc/default/grub

Add the following parameters to the GRUB_CMDLINE_LINUX_DEFAULT line:

# If using Intel:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

# If using AMD:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

Update the system and reboot:

sudo update-grub && sudo reboot

Step 2: Identify Network Card Capabilities

Find the name of your physical network interface (PF) using the ip link show command. Let’s assume the card name is enp1s0f0. Check the maximum number of Virtual Functions (VFs) it can create:

cat /sys/class/net/enp1s0f0/device/sriov_totalvfs

If the result is 32 or 63, you are ready for the next step.

Step 3: Create Virtual Functions (VFs)

Try creating 4 virtual network cards with the following command:

echo 4 | sudo tee /sys/class/net/enp1s0f0/device/sriov_numvfs

Use the lspci | grep Ethernet command to verify. You will see “Virtual Function” lines appear. To ensure this setting persists after a reboot, you should add the above command to /etc/rc.local or configure it via Netplan.

Step 4: Assign the Virtual Card to a KVM VM

Identify the PCI address of the VF using virsh nodedev-list | grep pci. Then, edit the VM’s configuration file:

virsh edit vm_name

Add this code block inside the <devices> tag (ensure you replace the bus and slot addresses with the correct ones for your VF):

<interface type='hostdev' managed='yes'>
  <source>
    <address type='pci' domain='0x0000' bus='0x01' slot='0x10' function='0x0'/>
  </source>
</interface>

Start the VM and check. You will see a genuine Intel network card inside, ready to handle high-speed traffic with the lowest possible latency.

Conclusion

Although configuring SR-IOV is more labor-intensive than a Linux Bridge, the benefits are well worth it. It frees up the host CPU for other tasks and allows VM networking to run as smoothly as a physical machine. If you are operating mission-critical services, try implementing SR-IOV today. You will see a significant difference in system performance!