Why Do “Beast” VMs Run Slow? Lessons from the Field
Have you ever assigned 32 vCPUs and 128GB of RAM to a SQL Server VM only to get inexplicably sluggish results? I once encountered a bizarre case: small VMs were running smoothly, while that “monster” machine responded very slowly. CPU Ready and Disk Latency stats looked perfect (all green), but in reality, performance had dropped by 30%. This is a critical step even if you optimize Windows virtual machines for better performance. The main culprit was a misaligned NUMA Topology.
When building large virtual machines (Wide VMs) where the number of vCPUs or RAM exceeds the capacity of a single physical socket, VMware’s default configuration is often no longer optimal. This is when we need to intervene in the Virtual NUMA (vNUMA) to get everything back on track.
Understanding NUMA: Don’t Let the CPU “Borrow” RAM
What is Physical NUMA?
Imagine a server with 2 Intel Xeon CPUs. Each CPU directly manages its own region of RAM, called Local Memory. If CPU 1 needs to fetch data from CPU 2’s RAM, it must go through a high-speed interconnect (like Intel’s UPI or AMD’s Infinity Fabric). Accessing this Remote Memory increases latency from 60ns to over 100ns, causing applications to stutter, much like the performance hits from RAM shortages.
The Role of Virtual NUMA (vNUMA)
vNUMA exposes this physical structure to the Virtual Machine (Guest OS). This allows Windows or Linux to know exactly which CPU is paired with which RAM region. The operating system will then prioritize scheduling processes within a single node, avoiding the need to “borrow” RAM across sockets, which creates a bottleneck.
How to Check the NUMA Topology on a Physical Host
Don’t guess; look at the specific numbers. You need to know exactly how many NUMA nodes your host has. The quickest way is to SSH into ESXi and type esxtop, then press m to view memory or perform vSphere diagnostics.
Alternatively, use a direct command in the ESXi Shell to get parameters quickly:
# Count the number of NUMA nodes on the host
esxcli hardware memory get | grep "NUMA Node Count"
# View detailed core allocation
localcli hardware cpu list | grep "Node ID" | sort -uSuppose the host has 2 Sockets, each with 16 cores. That means you have 2 physical NUMA nodes, each holding 16 cores.
Hands-on: How to Configure vNUMA Correctly?
By default, VMware automatically enables vNUMA when a VM has 9 or more vCPUs. However, this algorithm can sometimes be “out of sync” with modern CPUs like AMD EPYC, which have complex chiplet structures.
1. The “Alignment” Rule
The key point is: A VM’s vNUMA node must fit entirely within a single physical NUMA node.
If the host has 16 cores/socket but you create a 20 vCPU VM, VMware is forced to split the machine into 2 vNUMA nodes. If they are split unevenly (e.g., a 4 vCPU node and a 16 vCPU node), performance will fluctuate inconsistently.
2. Adjusting Cores per Socket (Note for vSphere 6.5+)
In the past, techs often adjusted Cores per Socket to force a vNUMA topology. Since version 6.5, VMware has become smarter by decoupling the display topology from the underlying vNUMA structure. However, this parameter is still crucial for software licensing, such as SQL Server Standard (which limits the number of recognized sockets).
- Advice: Leave
Cores per Socket = 1as the default. Only change it when software requires a socket limit, and ensure this number of cores is a divisor of the physical cores on a single node.
3. Fine-tuning Advanced Parameters
To become an optimization “wizard,” go to Edit Settings > VM Options > Advanced > Edit Configuration and pay attention to these two parameters:
numa.autosize.once = FALSE
The default is TRUE, meaning vNUMA is only calculated once when the VM is created. If you vMotion the VM to another host with a different CPU generation, the vNUMA will be misaligned. Setting this to FALSE helps the VM recalculate every time it boots to match the new hardware.
numa.vcpu.preferHT = TRUE
Use this when you want to pack a large VM into fewer NUMA nodes by utilizing threads (Hyper-Threading). For example: a 24 vCPU VM can fit into a single 16-core/32-thread physical node. RAM latency will drop significantly, even if pure computing power might decrease slightly.
Verifying the Results
After restarting, check if the Guest OS understands your intent. For Linux, use the command numactl --hardware. For Windows, download the Coreinfo tool from Sysinternals and run:
coreinfo.exe -nIf the vCPUs are grouped correctly and RAM is divided evenly among the nodes, you have succeeded.
Real-world Experience: SAP HANA Case Study
I once handled an SAP HANA system running on AMD EPYC CPUs. Initially, queries ran very slowly due to “Memory Slop.” After intervening with the numa.nodeCount parameter to match the exact number of AMD L3 Caches, query processing speed skyrocketed by 20%. The best part is that this result was achieved without spending a single cent on hardware upgrades.
Conclusion
vNUMA optimization isn’t for small VMs (under 8 vCPUs). But for Databases, ERP, or Big Data systems, it is a vital step. The more closely the vNUMA configuration matches the physical hardware, the lower the latency and the smoother the application runs. This process is just as vital as upgrading VM hardware version for modern environments. Good luck with your system optimization!
