PSOD: The Administrator’s Purple Nightmare
If Windows has the Blue Screen of Death, VMware ESXi has the Purple Screen, known as PSOD (Purple Screen of Death). I once handled operations for a major bank when a host carrying over 40 critical VMs suddenly “died.” Accessing the iDRAC, I was met with a solid purple screen covered in hex codes. The feeling at that moment was incredibly stressful.
In reality, a PSOD occurs when the VMkernel encounters a critical error it cannot recover from. Instead of continuing and risking data corruption, the system chooses to halt to protect data integrity. Mastering PSOD diagnosis helps you quickly distinguish between a faulty RAM stick and a driver conflict. This is a vital skill for minimizing business downtime.
Configuring Core Dumps: Don’t Let Your System “Die” in Silence
When a PSOD occurs, all data in RAM is written to the Core Dump partition. If you forget to configure this, all traces of the root cause will vanish as soon as you reboot. Don’t end up in a situation where the system hangs repeatedly without leaving a single log line behind.
1. Checking Existing Dump Partitions
Typically, ESXi creates this partition during installation. Use the following SSH command to check if your host already has a place to store its “last testament”:
esxcfg-dumppart -lIf the result is empty, you are running an extremely high-risk system. Set it up immediately.
2. How to Create a Dump File on a Datastore
For servers with ESXi installed on USB or SD cards, the system often lacks a default dump partition. You should create a dump file of about 2GB on a local disk (SSD/HDD) to ensure enough data is captured:
# Create a 2GB dump file on the Datastore
esxcli system coredump file add -d DATASTORE_NAME -f psod-dump-file
# Activate the newly created file
esxcli system coredump file set -p /vmfs/volumes/DATASTORE_NAME/vmkdump/psod-dump-file
# Reconfirm the status
esxcli system coredump file listExpert Tips for “Reading” the Purple Screen
Don’t rush to hit the Reset button. Take a photo of the screen or use your phone to record the entire content. There are 3 “golden” parameters you need to examine closely:
- Exception Type: For example,
Page Faultis usually memory-related, whileMachine Check Exception (MCE)is 99% likely a hardware failure. - Backtrace: This is the history of CPU functions executed before the crash. If you see module names like
lpfc(Emulex) ori40en(Intel), you’ve found the driver conflict culprit. - Server Uptime: If the server crashes after only a few minutes of operation, it’s likely a physical fault or a cooling issue.
Extracting Data After Reboot
After forcing a reboot, convert the dump file into a text format for deeper analysis. Don’t grope around in the dark.
# Extract logs from the dump partition
esxcfg-dumppart -L /vmfs/volumes/DATASTORE_NAME/vmkdump/analysis.log
# Quickly search for related errors
grep -i "coredump" /var/log/vmware/vobd.logPro-tip: Copy the first line of the PSOD error and search for it on the VMware Knowledge Base (KB). Most common driver issues already have official patches from the vendor.
Real-world Example: When Hardware “Speaks Up”
If you encounter a Machine Check Exception (MCE), prepare yourself to replace hardware. MCE is a signal from the CPU reporting uncorrectable hardware errors. In one real case, I checked the iDRAC logs after seeing an MCE and discovered that RAM stick #4 had an ECC (Error Correction Code) error. After replacing that stick, the system ran stably for the next two years.
Prevention is Better Than Cure: A 3-Step Process
To avoid losing sleep over PSODs, you must maintain strict operational discipline.
1. Synchronize Firmware and Drivers (HCL)
A classic mistake is updating ESXi while neglecting the BIOS, Network Card, or RAID Card. Mismatches between the OS driver and hardware firmware are “fertile ground” for PSODs. Always cross-reference with the VMware Compatibility Guide (HCL).
2. Centralized Monitoring via Syslog
Don’t wait for a purple screen to start looking for logs. Forward all logs to a centralized server like vRealize Log Insight or a Syslog Server. When a host “dies,” you still have the data on another server to dissect the cause.
# Configure log forwarding to a central server
esxcli system syslog config set --loghost='udp://192.168.1.100:514'
esxcli system syslog reload3. Simulation Testing (Lab Only)
Want to know how your system reacts when a crash occurs? You can intentionally trigger a simulated crash using the following command. Note: This command will crash the host immediately—never use it in Production.
vsish -e set /reliability/crashMe/Panic 1After migrating many systems from VMware to Proxmox, I’ve noticed that while Proxmox is flexible, ESXi’s Core Dump mechanism is still incredibly detailed. If you know how to leverage it, you will always be in control. Always keep your firmware up to date and don’t forget to take a screenshot when the screen turns “purple.”
