The Real-World Problem: HA Cluster “Running Fine” But Still Having Downtime
Last year, our vSphere cluster had 4 ESXi hosts running about 60 production VMs. One morning, a host suffered a power failure — completely dead. HA restarted the VMs onto the remaining hosts, but 8 VMs failed to start. The log reported “Insufficient resources to satisfy HA failover level for cluster”.
The HA cluster was actually doing exactly what it was supposed to — refusing to power on VMs because Admission Control was “reserving” resources for failover. But the configured threshold was too high relative to reality, creating a paradox: visually the cluster still had resources available, yet VMs were blocked from running.
That wasn’t all. DRS was set to aggressive mode — vMotioning VMs every few minutes at random, causing storage I/O to spike. The database VMs took the worst hit, with noticeable latency spikes every time they were migrated.
Root Cause Analysis: Two Misunderstood Mechanisms
Admission Control and the Default Slot Policy Trap
Admission Control operates on a simple principle: before allowing another VM to power on, it checks — if a host dies right now, does the cluster have enough CPU and RAM to restart all VMs? If not → block. Makes sense. The problem lies in how “enough” is calculated.
There are 3 calculation modes:
- Cluster resource percentage: Reserves X% of the cluster’s total CPU and RAM
- Slot policy (Legacy): Divides resources into equal “slots” and keeps N slots empty
- Dedicated failover hosts: Designates 1–2 hosts to always remain on standby, running no regular VMs
Our problem stemmed from the default Slot Policy. Slot size is calculated based on the VM with the highest reservation in the cluster — a SQL Server instance with a 32 GB RAM reservation. Result: each slot = 32 GB RAM. With 60 VMs, the number of slots that needed to be reserved became enormous, consuming nearly all “spare” resources even when the cluster actually had plenty of free RAM.
DRS Aggressive Mode: Constant vMotion Isn’t Always Good
DRS (Distributed Resource Scheduler) automatically load-balances via vMotion, with Migration Threshold ranging from 1 (Conservative) to 5 (Aggressive). At levels 4–5, DRS vMotions VMs even on minor imbalances — sometimes migrating a VM only to migrate it back a few minutes later. Database VMs without vMotion-aware workloads experience latency spikes every time they’re moved.
Solutions for Admission Control
Option 1: Cluster Resource Percentage (Best Fit for Most Environments)
Switch to percentage-based calculation of total resources. The formula is straightforward: if the cluster has N hosts and needs to tolerate 1 host failure, reserve 1/N × 100% of resources. With 4 hosts: reserve 25% CPU and 25% RAM.
# Connect to vCenter
Connect-VIServer -Server vcenter.lab.local -User [email protected]
$cluster = Get-Cluster -Name "Production-Cluster"
# View current Admission Control configuration
$cluster.ExtensionData.Configuration.DasConfig.AdmissionControlPolicy
# Update to Percentage-based: 25% CPU, 25% RAM
$spec = New-Object VMware.Vim.ClusterConfigSpecEx
$spec.dasConfig = New-Object VMware.Vim.ClusterDasConfigInfo
$spec.dasConfig.admissionControlPolicy = New-Object VMware.Vim.ClusterFailoverResourcesAdmissionControlPolicy
$spec.dasConfig.admissionControlPolicy.cpuFailoverResourcesPercent = 25
$spec.dasConfig.admissionControlPolicy.memoryFailoverResourcesPercent = 25
$spec.dasConfig.admissionControlPolicy.autoComputePercentages = $false
$cluster.ExtensionData.ReconfigureComputeResource($spec, $true)
Write-Host "Admission Control updated: 25% CPU, 25% RAM"Option 2: Dedicated Failover Host (If You Have a Spare Host)
No calculation needed — the simplest configuration. Designate one host to run no regular VMs, used only during failover. The obvious downside: wasted resources if that host sits idle 99% of the time.
$failoverHost = Get-VMHost -Name "esxi-04.lab.local"
$spec = New-Object VMware.Vim.ClusterConfigSpecEx
$spec.dasConfig = New-Object VMware.Vim.ClusterDasConfigInfo
$spec.dasConfig.admissionControlPolicy = New-Object VMware.Vim.ClusterFailoverHostAdmissionControlPolicy
$spec.dasConfig.admissionControlPolicy.failoverHosts = @($failoverHost.ExtensionData.MoRef)
$cluster.ExtensionData.ReconfigureComputeResource($spec, $true)Option 3: Disabling Admission Control — Not Recommended
Some teams disable it entirely for “more flexibility.” But when a real failover occurs, the cluster may not have enough resources to restart VMs — they’ll die permanently instead of restarting on another host. The risk isn’t worth it.
Optimizing DRS Migration Threshold
After testing each level in production over several weeks:
- Level 1–2: Almost no vMotion at all, imbalances persist. One host running at 80% while another sits at 30%.
- Level 3: VMware only vMotions on significant imbalance (>12% difference). Works well for most workloads.
- Level 4–5: Constant vMotion, high storage I/O, DB VMs are negatively impacted.
Level 3 is the sweet spot. More importantly, set VM Overrides for database VMs so DRS won’t migrate them automatically:
# Set DRS to Manual for latency-sensitive VMs
$vm = Get-VM -Name "sql-server-01"
$clusterSpec = New-Object VMware.Vim.ClusterConfigSpecEx
$vmOverride = New-Object VMware.Vim.ClusterDrsVmConfigSpec
$vmOverride.operation = [VMware.Vim.ArrayUpdateOperation]::add
$vmOverride.info = New-Object VMware.Vim.ClusterDrsVmConfig
$vmOverride.info.key = $vm.ExtensionData.MoRef
$vmOverride.info.enabled = $true
$vmOverride.info.behavior = [VMware.Vim.DrsBehavior]::manual
$clusterSpec.drsVmConfigSpec = @($vmOverride)
$cluster.ExtensionData.ReconfigureComputeResource($clusterSpec, $true)
Write-Host "$($vm.Name): DRS set to Manual — will not vMotion automatically"Best Approach: Combine Both and Monitor Regularly
Final configuration after 6 months of stable operation:
- Admission Control: Percentage-based, 25% CPU + 25% RAM
- DRS Level: 3 (Balanced)
- Database VMs: DRS Manual override
A daily health-check script to catch early signs of host overload:
# health-check-cluster.ps1
Connect-VIServer -Server vcenter.lab.local -User [email protected] -Password $env:VCENTER_PASS
$cluster = Get-Cluster -Name "Production-Cluster"
$hosts = Get-VMHost -Location $cluster
Write-Host "=== Cluster Health Check ===" -ForegroundColor Cyan
Write-Host "HA: $($cluster.HAEnabled) | DRS: $($cluster.DrsEnabled)"
foreach ($vmhost in $hosts) {
$cpuPct = [math]::Round(($vmhost.CpuUsageMhz / $vmhost.CpuTotalMhz) * 100, 1)
$memPct = [math]::Round(($vmhost.MemoryUsageMB / $vmhost.MemoryTotalMB) * 100, 1)
$warn = if ($cpuPct -gt 80 -or $memPct -gt 85) { "[WARN]" } else { "[OK] " }
Write-Host "$warn $($vmhost.Name) | CPU: $cpuPct% | RAM: $memPct%"
}
# Check for VMs with HA restart disabled
$dasConfigs = $cluster.ExtensionData.Configuration.DasVmConfig
$disabledCount = ($dasConfigs | Where-Object { $_.DasSettings.RestartPriority -eq 'disabled' }).Count
Write-Host "`nVMs with HA restart disabled: $disabledCount"Quick Comparison with Proxmox HA
When I later used Proxmox for a personal lab, the first thing that stood out: no Admission Control, no DRS. Create an HA Group, add VMs, done. Far simpler — a great fit for labs or small SMBs that don’t need fine-grained control.
VMware’s advantage is the ability to pin VMs to specific hosts, set reservations and limits, and let DRS auto-optimize within those boundaries. But Proxmox wins on visibility — HA logs are clearer, debugging is easier when VMs fail to restart. Digging through vSphere Event logs is… not a pleasant experience.
In summary, only two changes are needed: don’t leave Admission Control on the default Slot Policy when your cluster has VMs with high reservations — switch to Percentage-based with a threshold of 1/N × 100%. And set DRS to Level 3 with Manual overrides for latency-sensitive workloads. These two changes alone prevent a surprising number of unnecessary incidents.
