How to Run Nested VMware ESXi 8.0 on Proxmox VE: Build a vSphere Lab Without HCL Constraints

1. Background & Why Run Nested ESXi on Proxmox?

When preparing upgrade scenarios from vSphere 7.0 to 8.0 for production, many sysadmins hit an immediate roadblock: older lab servers have been dropped from VMware’s Hardware Compatibility List (HCL). ESXi 8.0 refuses to recognize Realtek RTL8111 NICs, drops support for Intel Xeon E5 v3/v4 processors, and halts right at the boot screen.

Investing in enterprise-grade servers just to test new features often costs between $1,500 and $3,000—an unnecessary expense for a lab environment. In this scenario, leveraging an existing Proxmox VE node to run Nested ESXi 8.0 is the optimal solution.

KVM on Proxmox acts as a hardware abstraction layer. It virtualizes the entire CPU with all necessary instruction flags, provides enterprise-grade VMXNET3/E1000e network adapters, and presents SCSI/NVMe controllers that the ESXi 8.0 installer recognizes seamlessly.

2. Prerequisites & Installing the ESXi 8.0 VM on Proxmox VE

First, you must ensure that the KVM kernel on the Proxmox host has nested virtualization enabled (Intel VT-x or AMD-V).

Enabling Nested Virtualization on the Proxmox Host

Open an SSH terminal to your Proxmox node and check the kernel module status:

# Check for Intel CPUs (a result of Y or 1 means enabled)
cat /sys/module/kvm_intel/parameters/nested

# Check for AMD CPUs
cat /sys/module/kvm_amd/parameters/nested

If the output returns N or 0, configure the modprobe file to enable it permanently:

# For Intel CPUs
echo "options kvm_intel nested=1" > /etc/modprobe.d/kvm-nested.conf

# For AMD CPUs
echo "options kvm_amd nested=1" > /etc/modprobe.d/kvm-nested.conf

# Reload module (note: ensure no VMs are running)
modprobe -r kvm_intel && modprobe kvm_intel || reboot

Creating a Standard ESXi 8.0 VM via CLI

Upload the VMware-VMvisor-Installer-8.0U2.iso ISO file to the local storage. Then, run the command to create the VM with standard parameters to prevent Purple Screen of Death (PSOD) errors:

qm create 900 --name nested-esxi-01 \
  --memory 16384 --balloon 0 \
  --cores 4 --sockets 1 --cpu host \
  --ostype other --bios ovmf \
  --scsihw virtio-scsi-single \
  --scsi0 local-lvm:60,format=raw,cache=writeback \
  --net0 vmxnet3,bridge=vmbr0,firewall=0 \
  --cdrom local:iso/VMware-VMvisor-Installer-8.0U2.iso \
  --boot order=cdrom;scsi0

3. Detailed Configuration: CPU Type, Storage & Promiscuous Network

If creating the VM via the Proxmox web interface, pay close attention to these three mandatory settings:

Configuring CPU Type = Host

This is the most critical parameter. If left at the default kvm64, the VMware installer will flag missing hardware flags and halt installation. Selecting host passes all instruction sets (VT-x/AMD-V, SSE4.2, AES-NI) directly from the physical CPU into the ESXi virtual machine.

Configuring Promiscuous Mode for Linux Bridge

Each nested guest VM running inside Nested ESXi carries its own MAC address. By default, the Linux Bridge (vmbr0) on Proxmox filters and drops packets with unknown MAC addresses, preventing nested VMs from obtaining DHCP IP addresses.

Quick fix: disable MAC learning on the bridge so packets are forwarded freely. Open /etc/network/interfaces on the Proxmox host:

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.10/24
    gateway 192.168.1.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
    bridge-ageing 0

The bridge-ageing 0 directive puts the bridge into free broadcast/unicast forwarding mode. After editing, run ifreload -a to apply the configuration.

Bypassing Legacy CPU Checks at Boot

On Intel Xeon Haswell/Broadwell generations or 4th–6th Gen Core i processors, the ESXi 8.0 installer may display an unsupported CPU warning. At the 5-second countdown bootloader screen, press Shift + O, append the following parameter to the end of the boot command line, and press Enter:

allowLegacyCPU=true

4. Verification, Guest VM Deployment & Resource Monitoring

Once installation finishes, reboot VM 900 and access the ESXi Host Client via web browser using the IP displayed on the console screen (e.g., https://192.168.1.50).

Creating a Test Guest VM for Network Verification

Upload a lightweight ISO (such as Alpine Linux ~50MB or Ubuntu Server 24.04) to the ESXi datastore. Create a test VM and power it on. Log into the guest VM to verify routing:

# Check IP address assigned via LAN DHCP
ip a

# Ping the default gateway and public DNS
ping -c 4 192.168.1.1
ping -c 4 1.1.1.1

If ping replies are stable with sub-millisecond latency, your nested network path is fully functional.

Monitoring System Load on the Proxmox Host

Nested virtualization typically introduces an additional 5–10% resource overhead. Important note: Never enable Memory Ballooning for an ESXi VM. The ESXi kernel requires static RAM allocation and will crash immediately if subjected to dynamic memory ballooning.

Check the actual VM load from the Proxmox CLI:

# View status and allocation details for VM 900
qm status 900 -v

# Monitor overall CPU and RAM load across the server
pvetop

With just a single physical Proxmox server featuring 64GB RAM and 8 CPU cores, you can comfortably deploy 2 Nested ESXi nodes alongside a vCenter Server Appliance (VCSA 8.0) VM. This provides the perfect testbed for vMotion, HA, DRS, and vSAN without spending your budget on dedicated enterprise hardware.