How to Use the vmkfstools Command on VMware ESXi: Managing, Cloning, and Converting VMDKs from the CLI

VMware tutorial - IT technology blog
VMware tutorial - IT technology blog

When the vSphere Client Fails You

A datastore hits 98% capacity in the middle of the night. A virtual machine suddenly freezes due to locked disk files. Meanwhile, vCenter displays only a vague error message: “An error occurred while communicating with the remote host”. If you’ve ever faced this scenario, you know the helpless feeling of being stuck with the graphical interface. The web GUI is great for day-to-day administration. However, when serious disk errors strike or vCenter loses connectivity, the ESXi CLI is the only place where you can safely troubleshoot.

In the 8-host ESXi cluster I manage, storage issues occur fairly often. The most common culprits are accidental Thick Provisioning depleting storage LUNs, or broken snapshot chains following a backup. During emergency troubleshooting sessions over SSH, the very first tool I turn to is always vmkfstools.

Decoding the VMDK Disk Structure and the Role of vmkfstools

vmkfstools is a low-level command-line utility running directly inside the VMkernel. It interacts straight with the VMFS file system and block storage devices without going through the vCenter abstraction layer.

What Does a VMDK Virtual Disk Actually Consist Of?

Administrators accustomed to Hyper-V or KVM often assume that a .vmdk file is a single encapsulated data block, similar to .vhdx or .qcow2. On ESXi’s VMFS file system, however, a virtual disk is split into two distinct files:

  • Descriptor File (vm-name.vmdk): An extremely lightweight plain ASCII text file, typically under 1 KB. It stores disk geometry parameters (cylinders, heads, sectors), the SCSI/NVMe adapter type, the virtual hardware version, and a pointer to the data file.
  • Flat File (vm-name-flat.vmdk): The binary file holding all actual operating system and user data. The web UI automatically hides the -flat suffix to keep things tidy. In the CLI, however, you will clearly see these two separate files.

Why You Shouldn’t Use the Standard cp Command

A risky habit among administrators new to ESXi is running cp disk1.vmdk disk2.vmdk. This practice frequently causes issues for two main reasons:

  • Ignoring sparse file mechanics: Standard Linux cp will inflate a 100 GB Thin disk (with only 10 GB of actual data) into a full 100 GB file, unnecessarily consuming physical capacity on your datastore.
  • Breaking file pointers: cp often copies only the descriptor file while missing the -flat.vmdk file entirely, or leaves internal pointers inconsistent.

vmkfstools resolves this with block-level read and write operations. It preserves dynamic allocation attributes and keeps the virtual disk structure intact.

Three Virtual Disk Formats You Need to Know

Whenever you create or convert disks using vmkfstools, you choose among three formats:

  • thin: Allocated on demand. The VMDK initially takes up only a few megabytes and expands dynamically as data is written, maximizing storage efficiency.
  • zeroedthick (Lazy Zeroed): Allocates all provisioned space upfront. ESXi zeroes out each block only when the virtual machine writes to it for the first time.
  • eagerzeroedthick (Eager Zeroed): Allocates and zeroes out every block immediately upon creation. Provisioning takes longer, but it delivers the most consistent I/O performance. This format is mandatory for Microsoft WSFC clusters and VMware Fault Tolerance (FT).

Hands-on Operations with vmkfstools

Preparation: Enabling SSH and Navigating to the Target Directory

By default, ESXi disables SSH for security. Access the ESXi Host Client, navigate to Manage > Services, right-click TSM-SSH, and select Start. Next, use PuTTY or your terminal to SSH into the host as root.

Navigate to the virtual machine’s folder on the datastore:

# View list of existing datastores
cd /vmfs/volumes/
ls -l

# Navigate to the affected virtual machine directory
cd /vmfs/volumes/datastore1/Database-Server/
ls -lh *.vmdk

1. Converting Thick Disks to Thin to Reclaim Storage Space

This is the scenario I encounter most frequently in production. A database VM was mistakenly provisioned with a 500 GB Thick disk, while actual data only takes up 80 GB—locking up over 400 GB of unused datastore space. First, power off the virtual machine.

# Syntax: vmkfstools -i [source-disk] [destination-disk] -d thin
vmkfstools -i Database-Server.vmdk Database-Server-Thin.vmdk -d thin

The system reports progress incrementally: Clone: 10% done., Clone: 20% done.… Once the clone completes, open the VM’s Edit Settings and point the virtual hard disk to the newly created Database-Server-Thin.vmdk. The datastore immediately reclaims 420 GB of free space.

2. Expanding Virtual Disk Size (Extend Disk) in Seconds

If you need to expand a virtual disk without waiting for the vCenter web interface to load, the -X flag gets it done instantly:

# Extend the disk to a total capacity of 150GB
vmkfstools -X 150G Database-Server.vmdk

Note that 150G represents the final target capacity, not the amount to add. If the current disk is 100 GB and you want to add 50 GB, the parameter value must be 150G.

For an eagerzeroedthick disk, specify the format flag so the system zeroes out the newly extended blocks immediately:

vmkfstools -X 150G -d eagerzeroedthick Database-Server.vmdk

3. Creating Standalone Virtual Disks from the Command Line

When you need to provision a secondary disk for a file server or set up database disk groups before attaching them to a VM:

# Create a 100GB Thin disk using an LSI Logic SCSI controller
vmkfstools -c 100G -d thin -a lsilogic DataDisk_01.vmdk

4. Recovering an Accidentally Deleted Descriptor File

During a routine VM directory cleanup, an administrator accidentally deletes the Database-Server.vmdk file. The VM immediately fails to boot with a “File not found” error. Fortunately, the underlying data file Database-Server-flat.vmdk is completely untouched. You can reconstruct the descriptor in three steps:

First, obtain the exact size of the flat file in bytes:

ls -l Database-Server-flat.vmdk
# Example output: 53687091200 bytes (equivalent to 50GB)

Next, use vmkfstools to generate a temporary disk matching the exact byte count and adapter type:

vmkfstools -c 53687091200 -a lsilogic -d thin temp.vmdk

Open the generated temp.vmdk file using the vi text editor:

vi temp.vmdk

Locate the flat file definition line (the sector count 104857600 is calculated automatically as 53687091200 / 512) and update it with the actual flat file name:

# Original line:
RW 104857600 VMFS "temp-flat.vmdk"

# Replace with the actual flat file name:
RW 104857600 VMFS "Database-Server-flat.vmdk"

Save the file, rename temp.vmdk to your original descriptor file name, and delete the temporary flat file:

mv temp.vmdk Database-Server.vmdk
rm -f temp-flat.vmdk

Power on the VM from the vSphere Client. The OS and all services will boot normally without losing a single byte of data.

5. Verifying Snapshot Disk Chain Consistency

Before consolidating snapshots or cloning critical VMs, check the disk chain consistency to prevent mid-operation failures:

vmkfstools -e Database-Server.vmdk

If the terminal outputs Disk chain is consistent, the snapshot chain and base disk structure are intact, and you can safely proceed with maintenance.

Conclusion

Mastering vmkfstools ensures you stay in control when storage problems strike your infrastructure. Command options like disk cloning with -i, disk expansion with -X, and descriptor reconstruction are essential troubleshooting skills for any VMware sysadmin. Rather than waiting idly or panicking over generic web UI error messages, a few precise CLI commands over SSH will quickly get your systems back online.

Share: