How to Configure KVM Nested Virtualization on Linux: Enable Nested Virtualization for Proxmox, VMware, and OpenStack Labs - ITFROMZERO - Share tobe shared! Encountering 'KVM acceleration not available' when installing Proxmox or ESXi inside a Linux VM? Here is how to configure KVM Nested Virtualization for nested labs.

Table of Contents

When Your Virtual Machine Refuses to Run VMs

To practice deploying a 3-node Proxmox VE cluster, experiment with VMware ESXi, or spin up an OpenStack DevStack lab, having several physical servers is ideal. In reality, few people have the budget or space for a full bare-metal rack at home.

The most popular workaround is using a mid-range Linux machine (around 8 cores, 32GB RAM) or a rented KVM VPS/server. You set up a Level-1 virtual machine (L1 Guest) and install Proxmox or ESXi on it. Trouble begins the moment you attempt to start a nested VM (L2) inside it:

TASK ERROR: KVM acceleration not available, using software emulation
# Or error message from VMware ESXi:
Hardware Virtualization is not a feature of the CPU, or is not enabled in the BIOS

The experience from here is excruciatingly painful. The VM falls back to software emulation (TCG), host CPU usage spikes to 100%, an Ubuntu Server image takes 15–20 minutes just to boot, or the ESXi installer halts the setup process entirely.

In my homelab (running a Ryzen 7 5700X with 64GB RAM), our entire test cluster for HA (High Availability) and Ceph storage runs inside nested VMs. In the past, I wasted an entire evening simply because I forgot to enable the virtualization flag on the host.

Why Can’t the L1 VM See Virtualization Extensions?

The nested virtualization model is divided into three distinct layers:

L0 (Bare-metal Host): The physical server running Linux with the KVM kernel module loaded.
L1 (Guest Hypervisor): The VM running on L0, acting as an intermediary hypervisor (e.g., Proxmox, ESXi).
L2 (Nested Guest): The VM running inside L1.

Intel CPUs provide the VMX (VT-x) instruction set, while AMD utilizes SVM (AMD-V). The L0 hypervisor leverages these instructions to assign hardware resources directly to virtual machines.

By default on many Linux distributions like RHEL or Debian, nested virtualization flags in the kvm_intel and kvm_amd kernel modules are disabled for memory safety. The L0 host masks these hardware flags rather than passing them through to L1.

Furthermore, when creating an L1 VM via libvirt with the default CPU model qemu64, the system only exposes a generic x86 instruction set. The L1 VM is stripped of all VT-x/AMD-V flags. Without these hardware extensions, L1 cannot initialize L2 guests.

Potential Approaches: Pros and Cons

There are three ways to address this issue, depending on your use case:

1. Pure Software Emulation (QEMU TCG)

L1 does not require /dev/kvm, handling all CPU instructions entirely in software.

Trade-off: Performance drops by 80–90% compared to hardware-assisted virtualization. This is only suitable for tinkering with lightweight Alpine Linux shells; running Kubernetes or Ceph clusters is out of the question.

2. Renting a Dedicated Bare-Metal Cluster

Rent three separate physical servers from providers like Hetzner or OVH.

Trade-off: You get 100% native performance and production-grade stability. However, monthly costs range from $150 to $300, which is far too expensive for personal homelab needs.

3. Enabling KVM Nested Virtualization with Host-Passthrough

Configure the L0 host to pass the native CPU instruction set directly to the L1 VM. This is the industry standard for staging environments, CI/CD pipelines, and DevOps/Sysadmin practice labs. The performance penalty for L2 compared to bare metal typically sits at only 5% to 15%.

Steps to Enable KVM Nested Virtualization

The deployment process involves four steps: verifying hardware support, enabling the kernel module, configuring the VM CPU model, and verifying from within the guest.

Step 1: Check Hardware Support on the L0 Host

Verify whether the physical CPU supports virtualization:

egrep -c '(vmx|svm)' /proc/cpuinfo

The returned number corresponds to your CPU thread count (e.g., 8, 16, or 32). If the output is 0, enter your motherboard’s BIOS/UEFI settings and enable Intel VT-x or AMD-V (SVM Mode).

Next, check the current status of nested virtualization on the host:

# For Intel CPUs:
cat /sys/module/kvm_intel/parameters/nested

# For AMD CPUs:
cat /sys/module/kvm_amd/parameters/nested

A return value of N or 0 means it is disabled. If it returns Y or 1, you can skip Step 2 and proceed directly to Step 3.

Step 2: Enable the KVM Module on the Host Kernel

Note: Shut down all running VMs on L0 before reloading the module. If any VM is active, running modprobe -r will fail with Module is in use.

# Temporarily apply for Intel CPUs:
sudo modprobe -r kvm_intel
sudo modprobe kvm_intel nested=1

# Temporarily apply for AMD CPUs:
sudo modprobe -r kvm_amd
sudo modprobe kvm_amd nested=1

To make this configuration persist across reboots, create a modprobe configuration file:

# For Intel CPUs:
echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm.conf

# For AMD CPUs:
echo "options kvm_amd nested=1" | sudo tee /etc/modprobe.d/kvm.conf

Verify again by reading /sys/module/kvm_intel/parameters/nested (or kvm_amd). Seeing Y or 1 confirms success.

Step 3: Configure CPU Passthrough for the L1 VM

With the host kernel configured, the next step is ensuring the L1 virtual machine recognizes the host CPU features.

For VMs managed via virsh (libvirt), open the XML configuration file:

virsh edit <L1-vm-name>

Locate the <cpu> block and update the attribute to mode='host-passthrough':

<cpu mode='host-passthrough' check='none'>
  <topology sockets='1' dies='1' cores='4' threads='1'/>
</cpu>

Equivalent steps using the virt-manager GUI:

Open the VM details window and navigate to CPUs.
Check Copy host CPU configuration (or enter host-passthrough in the Model field).
Click Apply and start the virtual machine.

Step 4: Verify Inside the L1 Virtual Machine

Power on the L1 VM, connect via SSH, and run the following checks:

# Check if virtualization flags are present
egrep -c '(vmx|svm)' /proc/cpuinfo

# Verify KVM hardware acceleration device
ls -l /dev/kvm

The count command should output a number greater than 0, and the character device /dev/kvm should exist (with access permissions for the kvm group). Your L1 virtual machine is now fully equipped to act as a hypervisor.

From here, you can deploy a Proxmox VE cluster, configure a 3-node HA setup, or install VMware ESXi to run smooth L2 virtual machines. Your entire complex lab infrastructure now runs neatly on a single physical host.