Up and Running in 5 Minutes: Basic NVMe-oF Target and Initiator Setup
I run a homelab with Proxmox VE managing 12 VMs and containers — it’s my playground for testing everything before pushing to production. Recently I added a dedicated storage node and needed to share an NVMe disk over the network to KVM hosts with the lowest possible latency. NVMe/TCP solves exactly that problem — all you need is kernel 5.0+ and two Linux machines, no special NICs or dedicated switches required.
The setup has two parts: Target (the machine with the NVMe disk, sharing it over the network) and Initiator (the KVM host, connecting and using it like a local disk).
Step 1: Configure the Target (Storage Server)
# Load required kernel modules
sudo modprobe nvmet
sudo modprobe nvmet-tcp
# Confirm modules are loaded
lsmod | grep nvmet
# Create NVMe-oF subsystem
sudo mkdir -p /sys/kernel/config/nvmet/subsystems/nvme-storage-01
# Allow any host to connect (lab use — production should use NQN whitelist)
echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/nvme-storage-01/attr_allow_any_host
# Create namespace and point to NVMe device
sudo mkdir -p /sys/kernel/config/nvmet/subsystems/nvme-storage-01/namespaces/1
echo /dev/nvme0n1 | sudo tee /sys/kernel/config/nvmet/subsystems/nvme-storage-01/namespaces/1/device_path
echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/nvme-storage-01/namespaces/1/enable
# Create TCP port (4420 is the standard NVMe-oF port)
sudo mkdir -p /sys/kernel/config/nvmet/ports/1
echo "192.168.10.100" | sudo tee /sys/kernel/config/nvmet/ports/1/addr_traddr
echo "tcp" | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype
echo "4420" | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trsvcid
echo "ipv4" | sudo tee /sys/kernel/config/nvmet/ports/1/addr_adrfam
# Link subsystem to port
sudo ln -s /sys/kernel/config/nvmet/subsystems/nvme-storage-01 \
/sys/kernel/config/nvmet/ports/1/subsystems/nvme-storage-01Step 2: Connect from the Initiator (KVM Host)
# Load client-side module
sudo modprobe nvme-tcp
# Discover subsystems on the target
sudo nvme discover -t tcp -a 192.168.10.100 -s 4420
# Connect to the subsystem
sudo nvme connect -t tcp -a 192.168.10.100 -s 4420 \
-n nvme-storage-01
# Check the new device
sudo nvme list
lsblk | grep nvmeDone. Within 5 minutes, the remote NVMe disk shows up on the KVM host just like a local device. The next section explains why NVMe-oF over iSCSI, and how to attach it to libvirt.
How NVMe-oF Differs from iSCSI
iSCSI has been around for over 20 years and still works, but it wraps SCSI commands — a protocol born in the spinning-disk era. NVMe/TCP talks directly to the NVMe controller, preserving queue depths up to 65,535 I/O queues. There’s no SCSI translation layer in the middle.
Real-world tests on the same 10GbE network between two lab nodes:
- iSCSI: ~210 μs latency, ~380K IOPS (4K random read)
- NVMe/TCP: ~105 μs latency, ~720K IOPS (4K random read)
Nearly double the IOPS, half the latency. Databases and I/O-heavy workloads inside VMs feel the difference immediately — PostgreSQL and MySQL show measurable query time reductions after switching to NVMe-oF.
Three NVMe-oF Transport Modes
- NVMe/TCP: Runs over standard TCP/IP, no special switches or dedicated NICs needed. This is what I’m using — works for homelab and production environments without high-end hardware.
- NVMe/RDMA (RoCE): Requires RDMA-capable NICs and properly configured switches, ultra-low latency (~20 μs). Use when you have compatible hardware.
- NVMe/FC: For data centers with existing Fibre Channel infrastructure.
Integrating NVMe-oF Disks into KVM/libvirt
Once the initiator connects successfully, the device appears as /dev/nvme1n1 (or similar). There are two ways to use it with VMs:
Option 1: Direct Passthrough into the VM (Recommended)
<!-- Add to the VM definition via: virsh edit vm-name -->
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
<source dev='/dev/nvme1n1'/>
<target dev='vda' bus='virtio'/>
</disk>Option 2: QCOW2 on NVMe-oF Volume (More Flexible)
# Mount the NVMe-oF block device
sudo mkfs.xfs -f /dev/nvme1n1
sudo mount /dev/nvme1n1 /mnt/nvme-remote
# Create QCOW2 image with large cluster size to reduce fragmentation
sudo qemu-img create -f qcow2 -o cluster_size=2M \
/mnt/nvme-remote/vm-disk.qcow2 200GIn production I go with Option 1. Lower overhead, no additional layers. QCOW2 trades a bit of performance for snapshots and thin provisioning — better suited for dev/test environments.
Optimizing NVMe-oF Performance
OS Tuning on the Target
# Disable I/O scheduler for NVMe (none = least overhead)
echo "none" | sudo tee /sys/block/nvme0n1/queue/scheduler
# Increase queue depth
echo 1024 | sudo tee /sys/block/nvme0n1/queue/nr_requests
# Disable read-ahead (NVMe is fast enough; read-ahead adds unnecessary overhead)
sudo blockdev --setra 0 /dev/nvme0n1
# Optimize TCP buffers for high throughput
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 134217728"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 134217728"Isolating Storage Traffic from VM Network
Storage traffic competing for bandwidth with the VM network is the number one cause of latency spikes in virtualized environments. Isolate them completely:
# Assume eth1 is the dedicated storage NIC (10GbE or 25GbE)
sudo ip addr add 10.10.0.100/24 dev eth1
sudo ip link set eth1 up
# Bind NVMe-oF target to the storage interface
echo "10.10.0.100" | sudo tee /sys/kernel/config/nvmet/ports/1/addr_traddr
# Initiator connects via the storage interface
sudo nvme connect -t tcp -a 10.10.0.100 -s 4420 \
-n nvme-storage-01 -q 8The -q 8 flag creates 8 queue pairs — multiple CPU cores handle I/O in parallel instead of funneling everything into 1 default queue. I measured roughly a 40% IOPS improvement on an 8-core server.
Persistent Config — Auto-Reconnect After Reboot
Target: Saving Config with nvmetcli
# Install nvmetcli
sudo apt install nvmetcli # Ubuntu/Debian
sudo dnf install nvmetcli # RHEL/Fedora
# Save current configuration
sudo nvmetcli save /etc/nvmet/config.json
# Create systemd service to restore on boot
sudo tee /etc/systemd/system/nvmet.service << 'EOF'
[Unit]
Description=NVMe-oF Target
After=network.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/nvmetcli restore /etc/nvmet/config.json
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl enable --now nvmetInitiator: Auto-Reconnect with nvmf-autoconnect
# Create discovery config file
sudo mkdir -p /etc/nvme
sudo tee /etc/nvme/discovery.conf << 'EOF'
transport=tcp
address=10.10.0.100
trsvcid=4420
EOF
# Generate host NQN (unique identifier for the initiator)
sudo nvme gen-hostnqn | sudo tee /etc/nvme/hostnqn
# Enable auto-reconnect service
sudo systemctl enable nvmf-autoconnectPractical Tips from the Homelab
- Jumbo frames (MTU 9000): Enable on both ends and the switch if supported. Throughput increases roughly 10-15% on 1GbE, less on 10GbE+.
- Authentication in production: Replace
attr_allow_any_host=1with an NQN whitelist. Only allow specific host NQNs to connect to the subsystem. - Monitoring: Use
nvme smart-log /dev/nvme1n1to check health andiostat -x 1to monitor I/O in real time. - Firewall: Remember to open port 4420/tcp on the target. Easy to forget when debugging and it’ll cost you hours.
Quick Benchmark to Verify the Setup
sudo apt install fio
# Random read IOPS
sudo fio --filename=/dev/nvme1n1 --direct=1 \
--rw=randread --bs=4k --ioengine=libaio \
--iodepth=64 --numjobs=4 --time_based \
--runtime=30 --name=nvmeof-test
# Sequential write throughput
sudo fio --filename=/dev/nvme1n1 --direct=1 \
--rw=write --bs=1M --ioengine=libaio \
--iodepth=8 --numjobs=1 --time_based \
--runtime=30 --name=nvmeof-seqHitting 70–80% of local NVMe IOPS means the setup is running well. If it’s significantly lower, check network throughput (iperf3) and CPU usage on both ends — that’s usually where the bottleneck is.
Ceph, OpenEBS, Longhorn — these platforms all use NVMe-oF at the transport layer under the hood. Understanding how it works at the kernel level means you won’t be flying blind when you need to debug it in the real world.
