Posted inVirtualization

Optimizing KVM Virtual Machine Performance with VirtIO Drivers and QEMU Tuning: Acceleration Techniques for Linux Guests

Virtualization tutorial - IT technology blog
Virtualization tutorial - IT technology blog

Context & Why It’s Needed: Don’t Let Your VMs Lag

Are you tinkering with KVM to build a homelab, or managing multiple critical Linux virtual machines on Proxmox VE? If so, you’ve probably noticed that virtual machines (VMs) run significantly slower than physical machines. I’m in the same boat – currently running 12 VMs and containers as a testing environment before deploying to production. This sluggishness is most noticeable in disk speed (Disk I/O) or network performance.

A bit about KVM (Kernel-based Virtual Machine): it’s a powerful virtualization solution that turns the Linux kernel into a hypervisor. KVM uses QEMU to emulate hardware. However, this emulation itself can affect performance. Imagine this: every time a virtual machine needs to access a disk or send/receive data over the network, it has to “ask permission” and wait for the hypervisor (QEMU) to perform that action. This leads to significant latency.

So, how can we make our virtual machines run fast, smoothly, almost on par with physical machines? The key solution lies in VirtIO Drivers and some QEMU Tuning techniques. VirtIO is a set of paravirtualized drivers that allow the guest OS (guest operating system) to communicate directly with the hypervisor efficiently, bypassing unnecessary hardware emulation layers of QEMU. This is like having a private highway instead of having to navigate winding country roads.

Installing VirtIO Drivers: The Foundation of Performance

The good news is that VirtIO drivers are already integrated into the kernel of most modern Linux distributions (such as Ubuntu, CentOS, Debian, Fedora). This means that when you install a new Linux operating system as a virtual machine, it will almost certainly automatically detect and use VirtIO if the hypervisor supports it.

Check if VirtIO is already in use

To be sure, you can check if VirtIO kernel modules are loaded in your Linux virtual machine using the following command:


lsmod | grep virtio

You will see modules like virtio_blk (for disk), virtio_net (for network), virtio_pci, virtio_console, etc. If this list appears, it means VirtIO is working correctly.

Check disk devices:


lsblk -o NAME,DISC-GRAN,DISC-MAX,DISC-ZERO,MOUNTPOINT,FSTYPE,MODEL,VENDOR

At this point, the disk name will start with vd (e.g., vda, vdb) instead of sd (SATA/SCSI) or hd (IDE). This is an indication that you are using a VirtIO Block device.

Check network card:


ip a

A VirtIO network card usually has a name starting with eth or enp. More importantly, it must be configured as VirtIO by QEMU. When creating a VM via virt-manager or Proxmox VE, always select `VirtIO` for both Disk and Network Device.

For existing virtual machines (legacy devices)

If you are using an older virtual machine with IDE/SATA/E1000 drivers, you should switch to VirtIO to maximize performance. This process requires caution, especially with the system disk. You need to add VirtIO drivers to initramfs/initrd before changing the disk type, ensuring the system can boot.

Step 1: Install and update initramfs (inside the VM)


# Ubuntu/Debian
sudo apt update
sudo apt install linux-image-extra-$(uname -r)
sudo update-initramfs -u -k all

# CentOS/RHEL/Fedora
sudo yum update
sudo dracut -f -v

Step 2: Shut down the VM and change device type (on Host or via interface)

With virsh (on host):


virsh edit <vm_name>

Find the disk and network card configuration section. Change from device='disk' / type='network' to use driver='qemu' and model='virtio' as follows:


<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='none' io='native'/>
  <source file='/var/lib/libvirt/images/yourvm.qcow2'/>
  <target dev='vda' bus='virtio'/>
</disk>

<interface type='bridge'>
  <mac address='52:54:00:xx:xx:xx'/>
  <source bridge='br0'/>
  <model type='virtio'/>
</interface>

After saving, restart the VM. If the virtual machine boots successfully, congratulations!

Detailed Configuration: Safely Overclocking KVM

While VirtIO is a crucial step, to achieve maximum performance, we need to perform deeper optimizations.

1. Optimizing VirtIO Block (Disk I/O)

Disk I/O performance is often the biggest “bottleneck.” With VirtIO, we can configure the following QEMU options to improve it:

  • Cache Mode: cache='none' and io='native'
    This is the most effective configuration I’ve found for both production and homelab environments. It allows the guest OS to manage disk cache directly, helping to avoid “double caching” and minimize overhead. Additionally, the `io=’native’` option uses Linux AIO (Asynchronous I/O), ensuring that I/O operations do not block each other.
    Configuration in the VM’s XML (`virsh edit` or Proxmox interfaceOrdinal):

    
<driver name='qemu' type='qcow2' cache='none' io='native'/>
  • Discard/TRIM Support:
    This feature allows the guest OS to notify the host when data blocks are no longer in use. This helps the host free up storage space while maintaining disk performance over time, especially crucial for SSDs.
    Add to the disk configuration in XML:

    
<driver name='qemu' type='qcow2' cache='none' io='native'>
  <discard enable='yes'/>
</driver>

    Inside the guest OS, you need to mount the filesystem with the discard option or run fstrim periodically.

    
# Check fstab
cat /etc/fstab
# You will usually see the discard option or fstrim.timer (systemd) being used.

2. Optimizing VirtIO Network (Network I/O)

  • Offloading Features:
    Features like TSO (TCP Segmentation Offload), GSO (Generic Segmentation Offload), LRO (Large Receive Offload) allow the virtual network card to handle some tasks that the CPU would normally perform. This helps reduce CPU load and increase network throughput.
    To check in the guest OS, use the command:

    
ethtool -k <interface_name>

    Ensure that tx-checksumming, tcp-segmentation-offload, generic-segmentation-offload, generic-receive-offload options are enabled ([fixed] or on). If not, you can try enabling them, although VirtIO usually handles this automatically:

    
sudo ethtool -K <interface_name> tso on gso on gro on
  • Multi-Queue Network:
    For VMs with multiple CPU cores, configuring multiple queues for the VirtIO network card helps distribute network I/O processing load across multiple cores. This significantly boosts performance for high-intensity network tasks.
    Configuration in the VM’s XML:

    
<interface type='bridge'>
  ...
  <model type='virtio'/>
  <driver name='vhost' queues='4'/> <!-- Number of queues equal to or less than the number of vCPUs -->
</interface>

    Then, inside the guest OS, you need to configure rps_cpus for each queue to assign them to specific CPUs. Create a script file and add it to a systemd service to run at boot:

    
#!/bin/bash

INTERFACE="eth0"
NUM_QUEUES=4 # Number of queues you configured in the VM's XML

# Get the path to the network card's RSS queue
QUEUE_PATH="/sys/class/net/$INTERFACE/queues/rx-0/rps_cpus"

# Assign each queue to a CPU core (e.g., 4 queues for the first 4 cores)
# Convert core number to bitmask: 0001, 0010, 0100, 1000...

for i in $(seq 0 $((NUM_QUEUES-1))); do
  CPU_MASK=$(printf "%%x" $((1<<i)))
  echo $CPU_MASK > "/sys/class/net/$INTERFACE/queues/rx-$i/rps_cpus"
  echo $CPU_MASK > "/sys/class/net/$INTERFACE/queues/tx-$i/xps_cpus"
done

# Enable Receive Packet Steering (RPS) and Receive Flow Steering (RFS) if needed
echo 1 > /proc/sys/net/core/rps_sock_flow_entries

3. VirtIO Balloon (Memory Management)

Ballooning allows the hypervisor to flexibly request or release memory from the guest OS. This feature helps utilize RAM more efficiently on the host.

If a VM doesn’t use all allocated RAM, the balloon driver will “shrink” its memory, freeing up RAM for other VMs.
Add to the VM’s XML:


<memorybacking>
  <balloon model='virtio'/>
</memorybacking&gt>

Note: While convenient, ballooning can cause unstable performance if the host is over-provisioned with too much RAM. Consider carefully when using it in environments that demand high and stable performance.

4. VirtIO RNG (Random Number Generator)

A high-quality entropy source is extremely important for cryptographic and security applications (like SSH, SSL/TLS). Virtual machines often encounter problems with insufficient entropy, slowing down these tasks. VirtIO RNG provides a way for the guest OS to directly obtain entropy from the host hypervisor.
Add to the VM’s XML:


<rng model='virtio'>
  <backend model='random'/> <!-- Or model='egd', depending on the host -->
</rng>

Inside the guest OS, install qemu-kvm-ev or rng-tools to utilize this entropy source:


# Ubuntu/Debian
sudo apt install rng-tools
sudo systemctl enable rngd
sudo systemctl start rngd

# CentOS/RHEL/Fedora
sudo yum install rng-tools
sudo systemctl enable rngd
sudo systemctl start rngd

# Check entropy
cat /proc/sys/kernel/random/entropy_avail
# This number should be high (above 1000) after rngd starts.

5. QEMU CPU Tuning

  • CPU Passthrough (host-model or host-passthrough):
    Instead of QEMU emulating a generic CPU, we can pass the host’s CPU information directly to the guest. This allows the guest OS to see and use all features of the physical CPU, including optimized instruction sets (like AVX, AES-NI) that QEMU might not fully emulate.
    Use host-model if you want easy live migration of VMs between hosts with similar CPUs. Use host-passthrough if you only run the VM on a specific host and need maximum performance.
    Configuration in the VM’s XML:

    
<cpu mode='host-model' check='partial'/> <!-- Or mode='host-passthrough' -->
  • Huge Pages:
    The Linux kernel typically uses 4KB memory pages. Huge Pages are larger memory pages (2MB or 1GB), which reduce the load on the CPU’s Translation Lookaside Buffer (TLB). This, in turn, improves performance for RAM-hungry applications. For database servers or heavy applications, Huge Pages can make a big difference.
    On the host (for example, with 16GB RAM, wanting to allocate 4GB for the VM using 2MB Huge Pages):

    
sudo sysctl vm.nr_hugepages=$((4096 * 1024 / 2048))
# The value is the number of huge pages. 4096MB / 2MB per page = 2048 pages.
# Add to /etc/sysctl.conf for permanent configuration.

sudo mkdir -p /dev/hugepages
sudo mount -t hugetlbfs none /dev/hugepages
# Add to /etc/fstab for automatic mount.

    In the VM’s XML:

    
<memorybacking>
  <hugepages/
</memorybacking>

Testing & Monitoring: Always Know Where You Stand

After optimization, it’s crucial to measure to see if the changes are effective. Don’t just rely on intuition; use benchmarking tools.

Measure Disk I/O Performance

My familiar tool is fio. It’s extremely flexible and allows simulating various types of I/O loads. Install and run it inside the guest VM:


# Ubuntu/Debian
sudo apt install fio

# CentOS/RHEL/Fedora
sudo yum install fio

# Example random read 4KB test
sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=testfile.img --bs=4k --iodepth=64 --size=1G --readwrite=randread --rwmixread=75 --numjobs=4 --runtime=60 --group_reporting

Compare the iops and bandwidth results before and after optimization. You’ll see a remarkable difference!

Measure Network Performance

iperf3 is the golden tool for testing network bandwidth between two points. You’ll need an iperf3 server (could be the KVM host or another VM) and an iperf3 client (the VM being tested).


# Install on both server and client
sudo apt install iperf3 # Ubuntu/Debian
sudo yum install iperf3 # CentOS/RHEL/Fedora

# On the server (server mode):
iperf3 -s

# On the virtual machine (client mode, replace <server_ip> with the server's IP):
iperf3 -c <server_ip> -P 4 -t 30 # -P is the number of streams, -t is the duration

Monitoring

Tools like htop, nmon, sar, iostat, mpstat are all useful for monitoring resources (CPU, RAM, Disk I/O, Network I/O) in real-time or collecting historical data. Pay attention to %wa (wait I/O) in htop or mpstat – if it’s high, it’s a sign of I/O congestion. Check the utilization of the disk with iostat to see if the disk is overloaded.

Conclusion

Optimizing KVM virtual machine performance is not a one-and-done task; it’s an ongoing process. It requires an understanding of how components interact, along with the ability to measure and fine-tune. With the VirtIO Drivers and QEMU Tuning techniques I’ve shared, you have powerful tools to “unleash” the full potential of your Linux KVM virtual machines.

From homelabs to small production environments, correctly applying these tips will help your system run much smoother, faster, and more stably. Remember, on the IT journey, every little optimization contributes to building a robust and efficient system!

Share:
Tags: