Optimizing KVM Networking with virtio-net Multiqueue and vhost-net: Hitting Full 10GbE Without CPU Bottlenecks

Real-World Problem: 10GbE Hardware, but VMs Only Pulling Just Over 2Gbps

Recently, I upgraded my Mikrotik switch and installed a pair of Intel X520-DA2 10GbE NICs on my Proxmox homelab cluster. The physical host runs 12 VMs for a staging lab. After plugging in the DAC cables, I ran an immediate test between the Proxmox host and my TrueNAS server: iperf3 reported a smooth 9.41 Gbps, exactly as expected.

The issue only surfaced when I spun up an Ubuntu 22.04 VM acting as an NGINX reverse proxy to handle traffic for the backend cluster. When running an iperf3 benchmark directly from a workstation to this VM via Linux Bridge (vmbr0), bandwidth stalled at around 1.8 to 2.2 Gbps. Slower throughput aside, checking htop on the host was alarming: vCPU 0 on the virtual machine was pinned at 100% load consumed entirely by ksoftirqd/0, while the remaining three cores sat virtually idle.

Why Does Default VirtIO Create a Bottleneck?

Digging in with mpstat -P ALL 1 and inspecting QEMU’s I/O stack revealed that the default virtual network adapter runs into two classic architectural bottlenecks:

1. The virtio-net Single-Queue Trap

By default, the virtio-net adapter only has a single TX/RX queue pair. This has two consequences:

• All network interrupts (IRQs) and incoming/outgoing packets are dumped onto a single CPU core to handle.
• Allocating 4, 8, or 16 vCPUs to the VM makes no difference. Once that interrupt-handling core hits 100%, total system throughput immediately throttles. At around 300,000 to 400,000 packets/sec (pps), a single queue will be exhausted almost instantly.

2. QEMU Userspace Context Switching Overhead

By default, the virtio-net backend runs in userspace via the QEMU process. As a result, the packet traversal path involves extra hops:

Host Kernel (Physical NIC / Bridge) → QEMU Userspace Process → KVM Guest Kernel.

Packets must constantly cross back and forth between kernel space and user space. This introduces massive context-switching overhead, driving up latency and wasting valuable CPU cycles.

Comparing 3 Common Approaches

When facing this bottleneck, infrastructure administrators generally weigh three options:

Option 1: PCI Passthrough or SR-IOV

Assign the physical network card or a Virtual Function (VF) directly to the virtual machine.

• Pros: Near bare-metal performance (9.8 Gbps) with extremely low latency.
• Cons: Sacrifices virtualization flexibility entirely. You lose snapshots, cannot perform live migrations to other nodes without matching hardware configurations, and the NIC must support SR-IOV.

Option 2: Enable the vhost-net Module in the Host Kernel

Offload packet processing from the QEMU userspace process directly to the vhost_net in-kernel driver on the host.

• Pros: Bypasses userspace hops, cutting host CPU overhead by 25–30% right away.
• Cons: Keeping a single queue still leaves vCPU 0 bottlenecked under heavy traffic or floods of small packets.

Option 3: Combine virtio-net Multiqueue and vhost-net

This is the most balanced solution between throughput and operational stability. We allocate multiple TX/RX queue pairs matching the number of vCPUs, while delegating the entire backend to vhost-net in the kernel.

• Pros: Distributes interrupts evenly across vCPUs, fully saturating the 10GbE link while preserving live migration and snapshot capabilities.

Step-by-Step Implementation Guide

Here are the practical steps on KVM/libvirt (or Proxmox VE) and inside the guest OS.

Step 1: Load the vhost_net Module on the Host

Open a terminal on your KVM host and check for the module:

# Check if the module is loaded
lsmod | grep vhost_net

# Load the module if not already loaded
sudo modprobe vhost_net

# Enable auto-loading on boot
echo "vhost_net" | sudo tee -a /etc/modules

Step 2: Assign Multiqueue and the vhost Backend to the VM

Shut down the virtual machine before making changes. Open the VM’s XML configuration using virsh edit:

virsh edit ubuntu-proxy

Locate the <interface> block connected to your bridge and add the driver name='vhost' queues='N' attribute. Rule of thumb: match the number of queues to the VM’s vCPU count (capping at 8 queues is recommended to avoid lock contention across CPU cores):

<devices>
  <interface type='bridge'>
    <mac address='52:54:00:1a:2b:3c'/>
    <source bridge='vmbr0'/>
    <model type='virtio'/>
    <!-- Declare the vhost driver and 4 queues for a 4-vCPU VM -->
    <driver name='vhost' queues='4'/>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
  </interface>
</devices>

Proxmox VE tip: You do not need to edit files manually. Shut down the VM, navigate to Hardware > Network Device > Edit, select the VirtIO NIC, check the Multiqueue box, and enter your vCPU count (e.g., 4).

Step 3: Enable the Queues Inside the Guest OS

Power on the virtual machine. Modern versions of Ubuntu, Debian, or RHEL typically detect multiple queues automatically. However, you should still verify with ethtool:

# Identify the network interface name inside the VM
ip -br link

# Check supported vs. active hardware queues (e.g., eth0)
sudo ethtool -l eth0

If the Combined value under Current hardware settings still displays 1, run the following command to enable all 4 channels:

sudo ethtool -L eth0 combined 4

Step 4: Balance Network Interrupts with irqbalance

Having multiple queues is not enough if Linux does not distribute interrupts across cores. Install irqbalance to automatically spread IRQs evenly across CPU cores:

sudo apt update && sudo apt install -y irqbalance
sudo systemctl enable --now irqbalance

To inspect interrupt distribution while running a network test, run:

watch -n 1 "grep virtio /proc/interrupts"

Now, the interrupt counters for each queue (virtio0-input.0, input.1, input.2, input.3) will increment concurrently across all 4 vCPUs instead of piling up on CPU 0.

Real-World Benchmark Results

Running a 4-stream benchmark with iperf3 -c 192.168.10.5 -P 4:

• Network throughput: Jumped from 2.1 Gbps to 9.42 Gbps, right against the theoretical ceiling of a 10GbE link using standard MTU 1500.
• VM CPU load: All 4 vCPUs evenly shared the load at approximately 25–30% for network interrupt processing. No more 100% ksoftirqd spikes causing packet drops.
• Host CPU load: Because vhost-net processes packets directly in the kernel, overall CPU usage by the host QEMU process dropped by more than 40%.