Virtualization That “Works” vs. Production-Ready Virtualization Are Two Very Different Things
I’ve seen plenty of KVM setups on production servers configured like this: SELinux set to permissive, firewall completely disabled, VMs using libvirt’s default NAT network virbr0. The reason? “Set it up fast, fix it later.” The problem is that “later” usually arrives at the worst possible moment — when a VM needs to get an IP from an internal DHCP server, during a security audit, or when services inside a VM aren’t accessible from outside despite the firewall looking correct.
My company still runs a few CentOS 7 servers, and the migration path to AlmaLinux has already been handled. In parallel, I rebuilt the entire KVM infrastructure from scratch on CentOS Stream 9 — proper bridge networking, correct SELinux labels, complete firewalld FORWARD rules. This article shares what I learned from that process, including a few instances where VMs silently failed to get an IP address and the culprit turned out to be a single missing firewall rule.
The Virtualization Stack: KVM, QEMU, Libvirt — What Each One Does
The most common issue after installation is not knowing which layer is responsible when something goes wrong. Understanding the stack upfront makes troubleshooting much faster:
- KVM: A Linux kernel module that turns an x86 CPU into a hypervisor. Handles only CPU and memory virtualization.
- QEMU: An emulator that handles I/O — disk, network cards, USB, and more. Combined with KVM, it delivers near-native performance.
- Libvirt: The management layer — provides a unified API, the
libvirtddaemon, and thevirshCLI. - Virt-Manager: A GUI frontend for libvirt that connects remotely via SSH — convenient for managing multiple VMs from your laptop.
CentOS Stream 9 is the upstream of RHEL 9. Whatever lands in CS9 today will appear in RHEL 9.x a few months later. Kernel 5.14.x, full KVM patches, long lifecycle — this stack is seriously supported.
Installing KVM and Related Tools
Step 1: Verify CPU Hardware Virtualization Support
# Check for vmx (Intel) or svm (AMD) flag
grep -E --color=auto 'vmx|svm' /proc/cpuinfo | head -3
# Or use virt-host-validate for a more comprehensive check
virt-host-validateNo output from the grep command? Go into BIOS and enable VT-x (Intel) or AMD-V first. Without hardware virtualization, KVM simply won’t run — there’s no workaround.
Step 2: Install the Virtualization Package Group
# Install the full virtualization package group
dnf install -y @virtualization
# Install Virt-Manager if the host has a GUI, or use it from a local machine
dnf install -y virt-manager virt-viewer
# VM disk image management tools
dnf install -y libguestfs-tools guestfs-toolsThe @virtualization package group pulls in qemu-kvm, libvirt, libvirt-client, virt-install, and all dependencies. One command, done.
Step 3: Enable libvirtd and Verify the KVM Module
systemctl enable --now libvirtd
systemctl status libvirtd
# The KVM module must be loaded
lsmod | grep kvm
# Expected output: kvm_intel (Intel) or kvm_amd (AMD) + kvmConfiguring Bridge Network — The Most Important Step
Libvirt’s default NAT network (virbr0) is only suitable for personal labs. VMs behind NAT can’t get IPs from an internal DHCP server, and service ports aren’t directly accessible from outside. Production needs bridge networking — VMs appear on the physical network like real machines, getting IPs directly from the router or switch’s DHCP server.
Creating a Bridge with nmcli
First, check the name of the physical interface (ens3, eth0, enp2s0, etc. — it varies by machine):
nmcli device statusAssuming the physical interface is ens3:
# Create bridge interface br0
nmcli connection add type bridge con-name br0 ifname br0
# Attach the physical network card to the bridge
nmcli connection add type ethernet slave-type bridge \
con-name br0-ens3 ifname ens3 master br0
# Configure static IP for the bridge (replace with your actual IP)
nmcli connection modify br0 \
ipv4.addresses "192.168.1.100/24" \
ipv4.gateway "192.168.1.1" \
ipv4.dns "8.8.8.8,8.8.4.4" \
ipv4.method manual
# Disable STP if there's no switch loop (speeds up bring-up)
nmcli connection modify br0 bridge.stp no
# Activate
nmcli connection up br0
nmcli connection up br0-ens3
# Verify
nmcli connection show --active
ip addr show br0Configuring firewalld for Bridge Traffic
This is the most commonly overlooked step. After creating the bridge, firewalld needs to know which zone the interface belongs to. More importantly, you need a FORWARD rule to allow traffic between VMs and the outside network:
# Assign br0 to a zone (public or trusted depending on your network trust level)
firewall-cmd --permanent --zone=public --add-interface=br0
# FORWARD rule — without this, VMs can't ping outside
firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 \
-i br0 -o br0 -j ACCEPT
# If VMs need NAT to reach the internet through the host (for bridges on an internal network without internet access)
firewall-cmd --permanent --zone=public --add-masquerade
firewall-cmd --reload
firewall-cmd --list-all --zone=publicThe most common issue after setting up a bridge: the VM gets an IP but can’t ping outside. Check the FORWARD rule immediately — that’s the cause in 9 out of 10 cases.
SELinux and KVM: Configure It Correctly Instead of Disabling It
I understand why people disable SELinux — when you’re in a hurry and it blocks something, setenforce 0 is the first instinct. But with a hypervisor the risk is fundamentally different: if a process escapes from a VM, SELinux is the last containment layer between the attacker and the entire host. Disabling it removes that protection. Reading the audit log takes less than 5 minutes.
SELinux Context for VM Disk Images
Libvirt stores images in /var/lib/libvirt/images/ by default — the virt_image_t context is already correct there. If you use a different directory:
# Example using /data/vm-images
mkdir -p /data/vm-images
semanage fcontext -a -t virt_image_t "/data/vm-images(/.*)?"
restorecon -Rv /data/vm-images
# Verify context
ls -Z /data/vm-imagesHandling SELinux Denials When Errors Occur
# View recent denials related to qemu-kvm
ausearch -c 'qemu-kvm' --raw | audit2why
# Create a policy module from the denial (instead of disabling SELinux)
ausearch -c 'qemu-kvm' --raw | audit2allow -M my-qemukvm
semodule -X 300 -i my-qemukvm.pp
# Useful SELinux booleans for KVM
setsebool -P virt_use_nfs 1 # If using NFS storage for VMs
setsebool -P virt_use_samba 1 # If using SambaCreating VMs and Managing with Virt-Manager
Running a headless KVM host is no problem. Virt-Manager from your laptop connecting via SSH is all you need:
# Run on a local machine with a GUI
virt-manager --connect qemu+ssh://[email protected]/systemWant to spin up multiple VMs at once — say, 10 VMs for a test environment — the CLI is much faster than clicking through the GUI one by one:
virt-install \
--name centos9-vm1 \
--memory 2048 \
--vcpus 2 \
--disk path=/var/lib/libvirt/images/centos9-vm1.qcow2,size=20,format=qcow2 \
--cdrom /tmp/CentOS-Stream-9-latest-x86_64-dvd1.iso \
--network bridge=br0 \
--os-variant centos-stream9 \
--graphics vnc,listen=127.0.0.1 \
--noautoconsoleCommonly Used virsh Commands
# List VMs
virsh list --all
# Start / Graceful shutdown / Force stop
virsh start centos9-vm1
virsh shutdown centos9-vm1
virsh destroy centos9-vm1
# View VM resource information
virsh dominfo centos9-vm1
# Snapshot before updating
virsh snapshot-create-as centos9-vm1 snap-before-update \
"Before system update" --disk-only
# Console access (requires serial console inside the VM)
virsh console centos9-vm1Quick Troubleshooting for Common Errors
Error: “error: Failed to connect socket to ‘/var/run/libvirt/libvirt-sock'”
# Check the service
systemctl status libvirtd
# User must be in the libvirt group
usermod -aG libvirt $(whoami)
# Log out and log back in, or:
newgrp libvirtError: VM Fails to Get an IP Address
# Is the physical interface attached to the bridge?
brctl show br0
# Check firewalld
firewall-cmd --list-all --zone=public
# Check libvirt logs for the cause
journalctl -u libvirtd --since "30 minutes ago" | grep -i 'error\|warn'Conclusion
Installing KVM on CentOS Stream 9 isn’t complicated. But the difference between a lab setup and a production setup comes down to three things: bridge networking instead of NAT, correct SELinux labels instead of disabling it, and firewalld FORWARD rules to allow VM traffic through. Together, these add only about 20 minutes compared to a default install, but save many hours of debugging down the line.
If you want to go further: add VLANs on the bridge to isolate traffic between VM groups, or switch to an LVM storage pool instead of file-based images to improve I/O performance. But that’s a story for after this foundation is solid.
