Guide to VFIO PCIe Passthrough on Linux KVM: GPU Passthrough for Windows VMs - ITFROMZERO - Share tobe shared! Configure VFIO PCIe Passthrough on Linux KVM to harness up to 98% of dedicated NVIDIA/AMD GPU performance in a Windows VM. Comprehensive guide covering vfio-pci driver binding, fixing Code 43, and CPU pinning to eliminate micro-stutter.

Table of Contents

Quick start: 5-Minute Setup

It’s 2 AM, and an urgent ticket arrives: a Windows VM rendering in Blender is bottlenecked hard. The virtual VirtIO display adapter is crawling at a meager 4 FPS in the viewport. To salvage the situation, the only viable solution is handing over a physical GPU directly to the VM via VFIO.

Here are the step-by-step instructions for Ubuntu 22.04 or Debian 12:

Step 1: Enable IOMMU in the bootloader

# Open GRUB configuration
sudo nano /etc/default/grub

# Add the appropriate CPU parameter to GRUB_CMDLINE_LINUX_DEFAULT:
# Intel CPU: intel_iommu=on iommu=pt
# AMD CPU:   amd_iommu=on iommu=pt
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt"

# Update GRUB and reboot
sudo update-grub
sudo reboot

Step 2: Identify the GPU Hardware IDs

lspci -nn | grep -E "VGA|Audio"
# Sample output for an RTX 3060:
# 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060] [10de:2504] (rev a1)
# 01:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)
# Note down the Vendor:Device ID pair: 10de:2504, 10de:228e

Step 3: Isolate the GPU with the vfio-pci driver

The Linux host must release the GPU before the VM boots. Create a configuration file to bind the vfio-pci driver to the IDs identified above:

echo "options vfio-pci ids=10de:2504,10de:228e" | sudo tee /etc/modprobe.d/vfio.conf

# Load required modules at boot
echo -e "vfio\nvfio_iommu_type1\nvfio_pci" | sudo tee -a /etc/modules

# Prevent default graphics drivers from claiming the GPU
echo -e "blacklist nouveau\nblacklist nvidia\nblacklist snd_hda_intel" | sudo tee /etc/modprobe.d/blacklist.conf

# Update initramfs and reboot
sudo update-initramfs -u -k all
sudo reboot

After the server reboots, run this command to verify:

lspci -nnk -d 10de:2504
# The 'Kernel driver in use' line must explicitly state: vfio-pci

Under the Hood: IOMMU and the IOMMU Group Puzzle

Standard virtual machines communicate solely through virtualized display adapters such as QXL or VirtIO GPU. The host CPU bears the full burden of translating graphics instruction sets. While this setup is smooth enough for text editing or web browsing, performance plummets by over 70% in 3D rendering or AAA gaming.

VFIO (Virtual Function I/O) eliminates this bottleneck completely. It grants the virtual machine direct control over the physical GPU’s PCIe registers, delivering 97–99% of bare-metal performance.

Understanding IOMMU Groups

An IOMMU functions as a dedicated MMU for I/O devices, mapping the VM’s virtual memory address space directly to the hardware card’s DMA bus. However, motherboard hardware does not isolate each PCIe slot individually; instead, it bundles them into IOMMU Groups.

KVM’s cardinal rule is straightforward: an IOMMU Group is the smallest unit of hardware isolation. You must assign all devices within the same group to the VM, or none at all. No exceptions.

Run the following script to inspect your device groups:

#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU Group %s ' "$n"
  lspci -nns "${d##*/}"
done | grep -i nvidia

If your graphics card and its onboard HDMI audio controller share Group 13, you must bind both to vfio-pci. Conversely, if the GPU is grouped alongside a USB controller or onboard NIC, the VM will not be able to boot independently.

Advanced Optimization: Fixing Code 43 and Reducing Frametime Latency

Successfully binding the card on the host is only half the battle. Boot up Windows, and the infamous yellow exclamation mark quickly strikes: Code 43.

1. Resolving Code 43 for Good

Starting with driver version 465 (April 2021), NVIDIA lifted hypervisor virtualization restrictions on GeForce cards. Nonetheless, many games and graphics applications still actively scan for hypervisors and refuse to run. The definitive fix is hiding KVM signatures inside the VM’s XML configuration (virsh edit win10-gpu):

<features>
  <acpi/>
  <apic/>
  <hyperv mode='custom'>
    <relaxed state='on'/>
    <vapic state='on'/>
    <spinlocks state='on' threshold='8191'/>
    <vendor_id state='on' value='GenuineIntel'/>
  </hyperv>
  <kvm>
    <hidden state='on'/>
  </kvm>
  <ioapic driver='kvm'/>
</features>

2. PCIe Passthrough and Clean VBIOS Patching

If the passthrough GPU is the host’s primary display adapter at boot, the host UEFI modifies its ROM shadow memory. This results in a persistent black screen when the VM attempts to initialize the display driver. Download a clean original VBIOS dump from TechPowerUp (or extract it beforehand), then reference it directly inside the <hostdev> tag:

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
  </source>
  <rom file='/var/lib/libvirt/vbios/rtx3060.rom'/>
  <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x01' slot='0x01' function='0x1'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
</hostdev>

3. CPU Pinning and HugePages: Eliminating Micro-Stutter

Sudden micro-stuttering stems from aggressive host CPU core hopping (context switching) and memory page fragmentation. Pinning physical core/thread pairs to specific vCPUs keeps frametime variances well below 2ms:

<vcpu placement='static'>8</vcpu>
<cputune>
  <vcpupin vcpu='0' cpuset='4'/>
  <vcpupin vcpu='1' cpuset='12'/>
  <vcpupin vcpu='2' cpuset='5'/>
  <vcpupin vcpu='3' cpuset='13'/>
  <vcpupin vcpu='4' cpuset='6'/>
  <vcpupin vcpu='5' cpuset='14'/>
  <vcpupin vcpu='6' cpuset='7'/>
  <vcpupin vcpu='7' cpuset='15'/>
  <emulatorpin cpuset='0,8'/>
</cputune>
<memoryBacking>
  <hugepages/>
</memoryBacking>

Configuring 2MB HugePages further slashes TLB cache misses from ~12% down to under 0.8%, ensuring rock-solid frame delivery.

Field Notes & Troubleshooting Tips

Poor IOMMU grouping by the motherboard: When the GPU is trapped in the same group as a SATA controller, a practical workaround is adding the pcie_acs_override=downstream,multifunction kernel parameter. This forces the kernel to split groups apart. The trade-off: it breaks hardware DMA security boundaries between VMs, so use it strictly in homelabs or development environments.
Black screen upon VM startup: Disable CSM (Compatibility Support Module) in your motherboard BIOS. Always create the VM using OVMF (UEFI) firmware paired with the Q35 chipset, steering clear of legacy SeaBIOS or i440fx.
Headless VM operation: Plug an HDMI dummy plug (around $2) into the card’s output port. Install a Sunshine server on the Windows VM and stream to your workstation using the Moonlight client. A 1Gbps LAN setup delivers ultra-smooth 2K 120Hz streaming with decoding latency below 3ms.
Resolving BAR 0 resource conflicts: If dmesg | grep -i vfio throws a BAR 0: cannot reserve error, Linux’s framebuffer is stubbornly clinging to the GPU. Fix this by appending video=efifb:off or initcall_blacklist=sysfb_init to your GRUB parameters.