Configuring Proxmox VE Replication with ZFS: High Availability and Disaster Recovery Without a SAN – ITFROMZERO

2 AM. The phone buzzes. Node 1 of my Proxmox cluster just died — a failed hard drive. 12 VMs were running on it, no shared storage, no SAN, nothing.

I’ve been in that situation. It wasn’t real production, thankfully just a homelab — but watching every VM go offline at once feels exactly the same. I run a homelab with Proxmox VE managing 12 VMs and containers — a playground to test everything before it hits production. And that night is when I started taking Proxmox VE Replication seriously.

Table of Contents

Why You Don’t Need a SAN for HA

Shared Storage (SAN/NFS) is the traditional HA solution: all nodes read and write to a common storage backend. If a VM dies on one node, another node restarts it immediately — zero downtime. But SANs are expensive, complex, and for homelabs or SMEs without the budget, they’re essentially not an option.

Proxmox VE Replication uses ZFS send/receive to synchronize VM disks between nodes on a configurable schedule (default: every 15 minutes). Here’s how it works:

Proxmox creates a ZFS snapshot on the source node
Sends an incremental snapshot to the target node over an encrypted SSH tunnel
The target node always holds the most recent copy of the VM disk (at most one interval behind)
When the source node dies → start the VM on the target node immediately, losing at most one sync cycle’s worth of data

This is not live migration and it’s not zero-downtime HA. But with a 15-minute RPO and an RTO measured in minutes rather than hours — at zero cost — this is a practical choice for most use cases where a SAN simply isn’t on the table.

Prerequisites

Make sure all of the following conditions are met. Missing any one of them will prevent Replication from working:

A Proxmox Cluster must already exist — at least 2 nodes joined to the cluster
A ZFS Pool on each node — VM disks must reside on ZFS storage, not LVM or directory storage
SSH key-based auth between nodes — Proxmox configures this automatically when creating a cluster; manual setup is usually not required
Sufficient internal bandwidth — the first sync copies the entire disk, so plan accordingly

Check ZFS pool and cluster status before proceeding:

# List available ZFS pools
zpool list
zpool status rpool

# Check cluster nodes
pvecm status
pvecm nodes

Verify that a VM’s disk is actually on ZFS (e.g., VM ID 100):

qm config 100 | grep -E "scsi|virtio|ide|sata"
# Correct output will show "local-zfs:" at the beginning:
# scsi0: local-zfs:vm-100-disk-0,size=32G

# If you see "local-lvm:" or "local:", migrate the disk to ZFS first

Step-by-Step Replication Setup

Creating a Replication Job via Web UI

The quickest way to get started:

Go to Datacenter → Replication → Add
Select the VM to replicate (by VM ID)
Select the Target — the destination node in the cluster
Choose a Schedule — the default */15 means every 15 minutes; adjust as needed
Click Create

Proxmox will kick off a sync immediately after creation. The first run takes a while since it transfers the entire disk — subsequent syncs are incremental and much faster.

Configuring via CLI for Multiple VMs at Once

When setting up in bulk or scripting the process, use the pvesr tool:

# Add a replication job: VM 100, target pve-node2, every 15 minutes
pvesr create 100-0 --vmid 100 --target pve-node2 --schedule "*/15"

# Replicate to a third node (second job for the same VM)
pvesr create 100-1 --vmid 100 --target pve-node3 --schedule "*/30"

# List all existing jobs
pvesr list

# Trigger a sync immediately without waiting for the schedule
pvesr run 100-0

The job ID format is <vmid>-<number>. A single VM can have multiple replication jobs pointing to different nodes — useful when you want to replicate to both node 2 and node 3.

Limiting Replication Bandwidth

This step is important and often skipped. Without a limit, the first sync (full copy) can saturate your internal network, directly impacting VMs running production workloads.

# Limit storage bandwidth (via web UI: Datacenter → Storage → local-zfs → Edit)
# Or edit /etc/pve/storage.cfg directly:
# bwlimit: 102400   # KB/s, i.e. 100 MB/s

# Alternative: limit at the network interface level with tc
tc qdisc add dev eth0 root tbf rate 200mbit burst 32kbit latency 400ms

Checking and Monitoring Replication

Viewing Real-Time Status

# Status of all jobs on the current node
pvesr status

# Sample output:
# VMID  STATE     SOURCE     TARGET     DURATION  FAIL
# 100   ok        pve-node1  pve-node2  0:00:23   0
# 101   ok        pve-node1  pve-node2  0:01:45   0
# 102   syncing   pve-node1  pve-node2  -         0

# View detailed logs for a specific job
journalctl -u pvesr@100-0 --since "2 hours ago" -f

Checking ZFS Snapshots on the Target Node

This is how you verify that data has actually reached the target node — don’t just trust the UI:

# SSH into the target node
ssh root@pve-node2

# List snapshots for the replicated VM
zfs list -t snapshot | grep vm-100

# Expected output — multiple incremental snapshots:
# rpool/data/vm-100-disk-0@__replicate_100-0__1700000000  1.2G
# rpool/data/vm-100-disk-0@__replicate_100-0__1700000900  45M
# rpool/data/vm-100-disk-0@__replicate_100-0__1700001800  38M

Real Failover Test — The Most Important Step

Replication you’ve never tested for failover might as well not exist. This is the hardest lesson I learned from that 2 AM incident — I had backups configured but had never tried a restore, and the moment I needed it I discovered the config was wrong.

Simulate a failover: migrate the VM to the target node, using the already-replicated ZFS dataset:

# Option 1: Via Web UI
# Select VM → More → Migrate → choose node → check "Allow Replication Overwrite"

# Option 2: CLI — migrate VM 100 to pve-node2, using the local disk already present
qm migrate 100 pve-node2 --with-local-disks 1

# The VM will use the latest ZFS snapshot on the target node,
# no need to copy from scratch — migration completes in seconds

Once the VM starts successfully on node 2, migrate it back to the original node:

qm migrate 100 pve-node1 --with-local-disks 1

Automated Monitoring Script

Create a script to run periodic checks and send an alert when a job fails:

#!/bin/bash
# /usr/local/bin/check-replication.sh

FAILED=$(pvesr status 2>/dev/null | awk 'NR>1 && $5 > 0 {print $1, "fail_count:", $5}')

if [ -n "$FAILED" ]; then
  MSG="[REPLICATION ALERT] $(hostname): $FAILED"
  # Send Telegram alert — replace YOUR_BOT_TOKEN and CHAT_ID
  curl -s -X POST "https://api.telegram.org/bot${BOT_TOKEN}/sendMessage" \
    -d "chat_id=${CHAT_ID}&text=${MSG}" > /dev/null
  exit 1
fi

echo "$(date): All replication jobs OK"
exit 0

chmod +x /usr/local/bin/check-replication.sh

# Run every 30 minutes
echo "*/30 * * * * root /usr/local/bin/check-replication.sh >> /var/log/pve-replication.log 2>&1" \
  > /etc/cron.d/proxmox-replication

Troubleshooting Common Errors

Error: “Could not connect to host”

# Test SSH from the source node to the target
ssh root@pve-node2 "echo OK"

# If it fails, refresh cluster certificates and SSH keys
pvecm updatecerts

# Check the cluster's authorized keys
cat /etc/pve/priv/authorized_keys

Error: “target storage does not exist”

This usually happens when the ZFS pool name or storage ID differs between nodes. Check the storage available on the target node:

pvesh get /nodes/pve-node2/storage
# Make sure the storage name matches what's configured in the replication job

Job Stuck in “syncing” State

# Kill the lock and retry
pvesr finalize 100-0 --lock-timeout 1
pvesr run 100-0

# If it still fails, check detailed logs
journalctl -u pvesr@100-0 -n 50

With this replication setup in place, I’m far more confident running production workloads on bare-metal Proxmox without shared storage. It’s not perfect HA like vSphere HA or Proxmox HA with shared storage — but at zero licensing and hardware cost, a 15-minute RPO/RTO is a completely acceptable trade-off, especially once you’ve tested the failover and know it works before you actually need it.