Posted inVMware

vCenter Frozen Due to Full /storage/db? A Definitive Guide for Sysadmins

VMware tutorial - IT technology blog
VMware tutorial - IT technology blog

When vCenter Suddenly “Goes on Strike” Due to a Full Disk

Monday morning, just as I took a sip of coffee, the company group chat exploded: “Hey, vCenter is down, reporting a 503 error!”. With an 8-host ESXi cluster carrying over 200 virtual machines, this was the signal for a long day ahead. After SSHing in to check, I found the /storage/db partition had reached 100%. This is a classic issue on the vCenter Server Appliance (VCSA) when the PostgreSQL database grows too quickly.

When the disk is full, the vpxd service automatically stops because it can no longer write data. Every operation, from VM management to backups, becomes completely paralyzed.

Why Does the /storage/db Partition Fill Up So Fast?

Based on real-world operational experience, I’ve identified 3 main causes:

  • Massive Tasks and Events data: Every action, such as a vMotion or a login, is recorded. For systems using automation scripts, this log table can grow by 2-5GB per week.
  • Performance Metrics: vCenter collects CPU and RAM data every 5 minutes. If you set a retention policy that is too long, the DB will run out of space very quickly.
  • Failed automatic cleanup mechanism: Sometimes the PostgreSQL VACUUM process doesn’t work efficiently, meaning old files aren’t released even after data is deleted.

Should You Clean Up Now or Expand the Disk First?

Faced with this incident, you usually have two options. Consider the pros and cons of each carefully.

Option 1: Data Cleanup

This approach addresses the root cause of data clutter and optimizes DB speed. However, it is quite time-consuming and risky if you aren’t proficient with SQL commands.

Option 2: Disk Expansion

This is the fastest “firefighting” method. vCenter will resume operation immediately without requiring deep internal configuration changes. The only downside is that it only treats the symptom; if the root cause isn’t found, the disk will fill up again in the future.

My advice: Expand the disk first to restore service, then take your time to clean up the data to optimize the system for the long term.

Steps to Check and Identify the Cause

Access the VCSA via SSH or Console. Switch to root privileges and type the following command:

shell
df -h

If you see /storage/db reporting 98% or 100%, we’ve diagnosed the problem correctly. To see which directory is taking up the most space, use the command:

cd /storage/db/vpostgres
du -sh * | sort -h

Typically, the data directory will account for the largest portion as it contains the actual database files.

Guide to Safely Expanding the /storage/db Partition

From version 6.7 onwards, VCSA has the ability to automatically detect new disks without requiring a reboot. This is a huge plus for VMware.

  1. Log in directly to the ESXi host running the vCenter VM.
  2. Right-click the vCenter VM and select Edit Settings.
  3. Find the disk corresponding to /storage/db (usually Hard Disk 8). Cross-check the current capacity to avoid mistakes.
  4. Increase the capacity (e.g., from 50GB to 100GB) and click OK.
  5. Return to the SSH window and run the command to automatically expand the LVM partition:
vpxd-servicecfg disk-expand

The system will scan the hardware configuration and automatically expand the capacity for you. Check again with df -h to confirm the results.

How to Clean Up Old Data in PostgreSQL

Once vCenter can “breathe” again, proceed to clean up the Tasks and Events tables. Note: Always take a Snapshot of vCenter before touching SQL!

Open the PostgreSQL manager:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

Check the top 10 largest tables using the following command:

SELECT nspname || '.' || relname AS "relation",
    pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
  FROM pg_class C
  LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
  WHERE nspname NOT IN ('pg_catalog', 'information_schema')
    AND C.relkind <> 'i'
    AND nspname !~ '^pg_toast'
  ORDER BY pg_total_relation_size(C.oid) DESC
  LIMIT 10;

If vpx_event or vpx_task exceeds 10GB, you can wipe the old logs using the TRUNCATE command:

truncate table vpx_event cascade;
truncate table vpx_task cascade;

Finally, run VACUUM FULL; to reclaim physical disk space. Note that this command will lock the tables, so only perform it when the service does not need to be active immediately.

Operational Best Practices for System Stability

To avoid having to stay up late troubleshooting, I apply these 3 rules:

  1. Tighten Retention Policy: Go to vCenter Settings -> Database Retention Policy. I usually only keep logs for 30 days instead of the default.
  2. Set Early Warnings: Create an Alarm in vCenter to notify you when disk usage exceeds 80%.
  3. Regular Native Backups: Use the built-in backup feature via FTP/SMB. If the DB is severely corrupted due to a full disk, restoring is the safest option.

Administering vCenter requires meticulousness, especially regarding data partitions. I hope these insights help you feel more confident when handling full disk errors on VCSA. If you encounter any difficulties at any step, don’t hesitate to leave a comment below.

Share:
Tags: