Posted inVirtualization

Rescue KVM/Proxmox VMs with libguestfs: Reset Root Password, Fix Boot Errors, and Recover Data Quickly

Virtualization tutorial - IT technology blog
Virtualization tutorial - IT technology blog

Comparing 3 Ways to Rescue KVM/Proxmox VMs on Boot Failure

Anyone managing KVM or Proxmox VE has encountered frustrating incidents: a VM hits a kernel panic right after a kernel upgrade, a mistyped UUID in /etc/fstab drops the system into emergency mode, or you simply forget the root password of a VM built six months ago. When the VM loses network connectivity and SSH is unreachable, you must intervene directly on the disk image.

Here are the 3 most common troubleshooting approaches:

  • Method 1: Attach a rescue ISO (SystemRescue, Ubuntu Live CD) and boot via VNC/Console. Familiar to beginners, but time-consuming.
  • Method 2: Use qemu-nbd to mount the disk image (qcow2/raw) onto the host. Allows reading and writing files directly from the host terminal, but carries a risk of LVM conflicts.
  • Method 3: Use the libguestfs tool suite (including guestfish, virt-edit, virt-customize, guestmount). Launches an isolated appliance kernel to operate directly on the virtual disk. Fast and safe.

Detailed Pros and Cons Comparison

1. Attaching a Rescue ISO

The biggest advantage is the familiar interface. You can perform chroot operations just like on a physical server.

Disadvantages:

  • Takes 5-10 minutes to download the ISO, attach it to the virtual hardware, and adjust the boot order.
  • Requires opening the noVNC Console to type commands manually, which often suffers from latency and keystroke lag.
  • Difficult to automate with bash scripts when dealing with 20-30 VMs at once.

2. Mounting Virtual Disks with qemu-nbd

Advantages: Works directly within the host shell without powering on the VM.

Disadvantages:

  • High risk of LVM Volume Group (VG) name collisions if both Host and Guest use default names like pve or ubuntu-vg.
  • Risk of filesystem corruption if the host kernel overwrites an incompatible partition table.
  • Cumbersome workflow: connect NBD, map partitions, mount, edit files, unmount, and disconnect.

3. The libguestfs Tool Suite

Advantages:

  • Runs independently on the host. No need to boot the VM and no reliance on hypervisor state.
  • Automatically detects LVM, software RAID, LUKS-encrypted partitions, and most common filesystems (ext4, XFS, Btrfs, NTFS).
  • Highly secure. Libguestfs spawns a temporary mini QEMU appliance running with isolated privileges, eliminating the risk of accidental host kernel interference.
  • Exceptional speed: reset passwords, modify configuration files, or copy data with a single command line in seconds.

Disadvantages:

  • The package pulls in several dependencies (QEMU mini kernel, supermin), taking up around 300MB – 500MB of disk space.
  • The VM must be COMPLETELY POWERED OFF (STOPPED). Modifying an active VM will result in data corruption.

Why libguestfs Saves Hours of Troubleshooting

In a Proxmox lab environment with 12 nodes and nearly 50 VMs, testing OS upgrades and modifying network configurations are daily routines. Previously, every time a VM failed to boot, mounting an ISO and logging into the console took 10 to 15 minutes per machine.

Switching to libguestfs condensed the entire process into a single CLI command on the host node. Recovery time dropped to under 30 seconds. The efficiency gain is most prominent when batch-fixing multiple VMs simultaneously.

Step-by-Step Implementation and Real-World Scenarios

Step 1: Install libguestfs

On Proxmox VE or Debian/Ubuntu:

apt-get update
apt-get install -y libguestfs-tools

On RHEL, Rocky Linux, or AlmaLinux 8/9:

dnf install -y libguestfs-tools

Kernel issue fix on Debian/Ubuntu: If you encounter the error supermin: error: failed to find a suitable kernel due to restricted kernel read permissions, run:

chmod 0644 /boot/vmlinuz-*

Step 2: Locate the VM Disk Path

First, ensure the VM is stopped (status: stopped). On Proxmox VE, disk images typically reside in one of the following formats:

  • ZFS Storage: /dev/zvol/rpool/data/vm-100-disk-0
  • LVM-Thin Storage: /dev/pve/vm-100-disk-0
  • Directory Storage (Qcow2/Raw): /var/lib/vz/images/100/vm-100-disk-0.qcow2

Step 3: 4 Common Rescue Scenarios

1. Reset Root Password in 5 Seconds

Use virt-customize to set a new password for the root user without booting the OS:

virt-customize -a /var/lib/vz/images/100/vm-100-disk-0.qcow2 --root-password password:NewPassword@2026

To create an additional emergency administrative user with sudo privileges:

virt-customize -a /var/lib/vz/images/100/vm-100-disk-0.qcow2 \
  --run-command 'useradd -m -s /bin/bash rescueadmin' \
  --password rescueadmin:password:AdminPass@2026 \
  --run-command 'usermod -aG sudo rescueadmin'

2. Fix Broken Configuration Files with virt-edit

Common scenario: you attach a second hard drive, enter an incorrect UUID in /etc/fstab, and Linux refuses to boot. The virt-edit command opens that file directly in your preferred host editor (nano or vi):

virt-edit -a /var/lib/vz/images/100/vm-100-disk-0.qcow2 /etc/fstab

Simply remove or correct the faulty line and save. The virtual disk data synchronizes immediately.

3. Inspect Boot Logs and Extract Emergency Data

Quickly view the last 50 log lines to diagnose crash causes:

virt-cat -a /var/lib/vz/images/100/vm-100-disk-0.qcow2 /var/log/syslog | tail -n 50

If the operating system is severely corrupted and you need to urgently extract source code or databases, mount the entire filesystem to a temporary host directory:

mkdir -p /mnt/vm_rescue
guestmount -a /var/lib/vz/images/100/vm-100-disk-0.qcow2 -i --ro /mnt/vm_rescue

Note: The -i (inspector) flag automatically mounts the correct root filesystem structure, while --ro (read-only) preserves original data integrity. You can now safely copy your data using rsync or cp.

Once backup is complete, unmount the directory:

guestunmount /mnt/vm_rescue

4. Access an Interactive Rescue Shell with virt-rescue

When you need to rebuild initramfs or reinstall the GRUB bootloader, enter the interactive rescue environment:

virt-rescue -a /var/lib/vz/images/100/vm-100-disk-0.qcow2

The system will provide a rescue shell where you can chroot and perform repairs:

# Run the following commands inside the virt-rescue shell:
mount /dev/sda1 /sysroot
chroot /sysroot
update-initramfs -u -k all
update-grub
exit

Two Essential Safety Rules to Remember

  • Never run libguestfs on a running VM: Concurrent writes by both the hypervisor and libguestfs will corrupt the partition table and cause permanent data loss.
  • Always snapshot before editing: Even though CLI operations are convenient, quickly create a Proxmox snapshot (or back up the qcow2 file) before executing invasive changes.
Share:
Tags: