Posted inVMware

VMware vSphere DirectPath I/O Configuration Guide: PCIe Device Passthrough for Virtual Machines

VMware tutorial - IT technology blog
VMware tutorial - IT technology blog

Heavy workloads are always a major challenge on virtualization platforms. When running AI training, large databases, or 10GbE/40GbE firewall appliances, the latency introduced by the ESXi hypervisor intercepting I/O operations often causes a noticeable drop in throughput. On the 8-host ESXi cluster I used to manage, we frequently hit CPU bottlenecks when handling hundreds of thousands of packets per second through standard vSwitches. DirectPath I/O is the ultimate key to breaking through these performance limits.

Three Approaches to Assigning Hardware to Virtual Machines

Depending on your resource demands and budget, you have three primary options:

  • Standard Virtualization (Emulated/Para-virtualized): ESXi provides virtual adapters such as VMXNET3 or LSI Logic SAS. Every I/O instruction passes through the host CPU for translation. While this is the most flexible approach, it places significant CPU overhead on the host under heavy loads.
  • Hardware Sharing (vGPU / SR-IOV): Physical hardware is divided into multiple Virtual Functions (VFs) shared across several VMs. However, this comes with costly licensing fees (such as NVIDIA AI Enterprise) and requires specialized hypervisor drivers.
  • VMware DirectPath I/O (PCIe Passthrough): Grants exclusive, direct control of a PCIe controller to a single virtual machine using IOMMU (Intel VT-d or AMD-Vi). The hypervisor is completely bypassed in the data path.

Trade-offs of Using DirectPath I/O

High performance always comes with trade-offs. You should carefully weigh the operational constraints before enabling it.

Key Advantages

  • Near Bare-Metal Performance: I/O latency drops from tens of microseconds to nanoseconds by entirely bypassing the ESXi kernel buffering layer.
  • Maximum Bandwidth: Take full advantage of native hardware throughput (e.g., 9.4 Gbps line-rate on 10GbE NICs) or millions of IOPS on PCIe 4.0 NVMe drives.
  • Zero Licensing Costs: A built-in feature available in both free ESXi and paid vSphere Standard/Enterprise Plus editions.

Operational Limitations to Plan For

  • Loss of vMotion and DRS: The VM becomes tethered to the physical host hosting the PCIe card. Live migrations during host maintenance are not possible.
  • No Snapshot or Fault Tolerance (FT) Support: ESXi cannot capture memory states or maintain lockstep synchronization for passthrough devices.
  • Mandatory 100% Memory Reservation: Memory overcommit cannot be used. Every megabyte of RAM assigned to the VM must be fully locked and backed by physical host memory.

Real-World Use Cases

In production environments, DirectPath I/O delivers the most impactful gains in three scenarios:

  1. VM Routers/Firewalls (pfSense, OPNsense, VyOS): Passing through Intel X520/X710 10GbE NICs enables line-rate routing without overwhelming host CPUs with network interrupts.
  2. AI & Data Science VMs: Dedicating an entire NVIDIA RTX 4090 or A100 GPU to run internal LLMs for data teams, saving tens of thousands of dollars in vGPU licensing.
  3. Storage Servers (TrueNAS Core/SCALE): Passing through an LSI SAS HBA in IT mode allows TrueNAS to directly query SMART data and manage ZFS storage pools on individual physical drives.

Step-by-Step Guide to Configuring DirectPath I/O on ESXi

Step 1: Enable IOMMU in Server BIOS/UEFI

Boot the server into BIOS/UEFI and enable hardware virtualization features:

  • Intel Systems: Enable Intel Virtualization Technology (VT-x) and Intel VT for Directed I/O (VT-d).
  • AMD Systems: Enable AMD-V and AMD IOMMU.

Step 2: Identify the PCI Device ID via SSH

Open an SSH terminal to your ESXi host to locate the hardware bus address:

# List network devices, graphics cards, and storage controllers
lspci | grep -iE "nvidia|ethernet|storage|lsi"

Take note of the device bus address, for example, 0000:03:00.0 for an Intel X520-DA2 network adapter.

Step 3: Enable Passthrough in vSphere Client

  1. Log in to the vSphere Client or ESXi Host Client.
  2. Navigate to Host > Configure > Hardware > PCI Devices.
  3. Click Toggle Passthrough.
  4. Find the target device using the PCI ID from Step 2, select its checkbox, and click OK.
  5. Reboot the ESXi host if prompted to release native kernel drivers.

Step 4: Attach the PCIe Device to the Virtual Machine

  1. Power off the target virtual machine.
  2. Right-click the VM > select Edit Settings.
  3. Click Add New Device > choose PCI Device.
  4. Select the passthrough device configured in Step 3 from the drop-down menu.
  5. Expand the Memory section and check Reserve all guest memory (All locked). This setting is strictly required for stable DMA (Direct Memory Access).
  6. Save the configuration and power on the virtual machine.

Step 5: Verify Device Recognition on the Guest OS

Log in to the Linux guest VM and verify the driver binding:

# Check if the guest OS detects the PCIe device
lspci -nnk | grep -iA 3 "ethernet\|vga\|3d\|storage"

# Confirm that the kernel loaded the native vendor driver (e.g., ixgbe, nvidia)
dmesg | grep -iE "pci|iommu"

If the device appears with its native vendor driver loaded, your setup is complete. The virtual machine can now operate with the full bare-metal performance of your hardware.

Share:
Tags: