Configuring Scratch Partition on VMware ESXi: The Ultimate Fix for Missing Logs and Core Dumps – ITFROMZERO

Table of Contents

Quick 5-Minute Fix via vSphere Client

If you are struggling with the warning “System logs are stored on non-persistent storage” or if your host is installed on an SD Card and loses all error traces after every reboot, follow these steps immediately:

Log in to the vSphere Client.
Select the ESXi host you want to configure and go to the Configure tab.
Navigate to the System section and select Advanced System Settings.
Click Edit and filter for the keyword ScratchConfig.ConfiguredScratchLocation.
In the Value field, enter the directory path on your Datastore. For example: /vmfs/volumes/HDD_DATA_01/.locker-ESXi01.
Click OK and Reboot the host to apply the changes.

Important Note: You must use the Datastore Browser to create the directory (e.g., .locker-ESXi01) before entering the path. If the directory does not exist, ESXi will ignore the setting and revert to using the Ramdisk.

Why You Shouldn’t Ignore the Scratch Partition

In real-world environments, I’ve seen many systems running for years with default settings. Once, while managing a cluster of 8 Dell R740 hosts with ESXi installed on 32GB SD cards, a host suffered a PSOD (Purple Screen of Death). When I tried to find the cause, the logs were empty because the Scratch Partition hadn’t been configured.

When you install ESXi on a large local hard drive, the system automatically creates a 4GB partition for Scratch. This serves as the “home” for:

System Logs: Operational logs used for troubleshooting.
Core Dumps: Critical data generated when a host crashes.
Temporary Files: Temp files used during patch installations (VIBs).

However, if you use a USB drive or SD Card, ESXi moves the Scratch location to a Ramdisk (/tmp/scratch) to protect the lifespan of the flash media. The Ramdisk has a maximum capacity of about 512MB and is wiped clean during a power loss or reboot. To maintain stability, it’s also vital to configure a graceful shutdown on power failure to prevent VM corruption.

CLI Configuration for Power Users

Using SSH is often faster and more accurate, especially when you need to copy-paste settings across multiple hosts. This is the workflow I typically use for new deployments.

Step 1: Get the Exact Datastore UUID

Using the Datastore name can sometimes fail if it contains spaces. Using the UUID is the safest method.

ls -l /vmfs/volumes/

You will see a string of IDs like 5eb1a2f3-7c8d9e0f.... Copy this string.

Step 2: Create a Unique Directory for the Host

Each host needs its own directory to prevent logs from overwriting each other.

mkdir /vmfs/volumes/5eb1a2f3-7c8d9e0f-1a2b-3c4d5e6f7a8b/.locker-Host01

Step 3: Assign the New Scratch Path

esxcli system settings advanced set -o /ScratchConfig/ConfiguredScratchLocation -s "/vmfs/volumes/5eb1a2f3-7c8d9e0f-1a2b-3c4d5e6f7a8b/.locker-Host01"

After running the command, type reboot for the host to recognize the new configuration immediately.

Automation with PowerCLI for Large Scale Management

When I worked on a project for a bank with over 50 hosts, I couldn’t manually click through each one. The PowerCLI script below helps you create directories and set configurations automatically, moving your workflow toward Infrastructure as Code.

# Connect to vCenter
Connect-VIServer -Server vcenter.yourdomain.com

$DatastoreName = "SAN-LOG-DATASTORE"
$Hosts = Get-VMHost

foreach ($VMHost in $Hosts) {
    $Directory = ".locker-" + $VMHost.Name
    New-DatastoreItem -Datastore $DatastoreName -ItemType Directory -Path $Directory
    
    $ScratchPath = "/vmfs/volumes/$DatastoreName/$Directory"
    Set-VMHostAdvancedConfiguration -VMHost $VMHost -Name "ScratchConfig.ConfiguredScratchLocation" -Value $ScratchPath
    Write-Host "Completed host: $($VMHost.Name)" -ForegroundColor Cyan
}

Real-world Experience: Prevention is Better Than Cure

Here are three lessons I’ve learned from many late-night troubleshooting sessions:

1. Never share folders: If you point two hosts to the same directory, the vmksummary.log file will be constantly overwritten. This results in corrupted logs that are useless for debugging.

2. Prioritize Local Storage: If the server has local SSDs, place the Scratch partition there instead of on the SAN. If the storage network (iSCSI/Fiber Channel) fails, leading to APD and PDL errors, the host can still record the error logs. If it’s on the SAN and the SAN goes down, you’ll have no idea what happened.

3. Plan for Capacity: Logs are small, but Core Dumps can take up several GBs. Ensure the Datastore has at least 20GB of free space. You certainly don’t want log files filling up a Datastore and affecting other Virtual Machines (VMs).

Conclusion

Configuring the Scratch Partition is a basic task, but it demonstrates the professionalism of an administrator. It allows you to be proactive when system issues arise, similar to configuring vSphere alarms for real-time monitoring. If you are managing an ESXi cluster, you can also use Host Profiles to ensure these settings are consistent across all nodes!