Configuring ZFS L2ARC Cache and SLOG on Proxmox VE: Accelerate VM I/O

Virtualization tutorial - IT technology blog
Virtualization tutorial - IT technology blog

Quick start: Adding Cache to a ZFS Pool in 5 Minutes

For those who already have a spare SSD/NVMe drive and want to attach it directly to Proxmox VE right away.

Open the Web Shell or SSH into your Proxmox host, then retrieve the list of disk IDs:

# List disks by disk-by-id (must use ID instead of /dev/nvme0n1 to avoid name shifts across reboots)
ls -la /dev/disk/by-id/

# Check the current pool name
zpool status

Assuming your ZFS pool is named rpool. You have prepared one partition for read caching (L2ARC) and one partition on an SSD with power loss protection (PLP) for SLOG. Run the following commands in sequence:

# 1. Attach the read cache (L2ARC)
zpool add rpool cache /dev/disk/by-id/nvme-INTEL_MEMPEK1W016GA_...-part1

# 2. Attach the separate intent log (SLOG)
zpool add rpool log /dev/disk/by-id/nvme-INTEL_MEMPEK1W016GA_...-part2

# 3. Verify the pool status
zpool status rpool

At this point, zpool status will list two additional branches, logs and cache, right below the main vdev. The pool is now ready for new workloads.

Technical Deep Dive: How Do L2ARC and SLOG Work?

In my homelab, a Proxmox cluster hosts 12 VMs and containers running CI/CD, Nextcloud, and PostgreSQL databases. Running such concurrent workloads on an HDD RAIDZ2 array or standard SATA SSDs can easily cause I/O wait to spike to 25–40%, leading to sporadic system freezes. Understanding the underlying mechanisms will help you avoid wasting money on pointless hardware upgrades.

1. L2ARC (Level 2 Adaptive Replacement Cache)

ZFS always prioritizes reading from RAM first. This memory tier is called the ARC. RAM has ultra-low latency (under 100ns), but capacity is limited and cost is high. When frequently accessed data begins to overflow RAM, ZFS evicts it down to storage disks.

Adding an SSD as an L2ARC introduces a second caching tier:

  • RAM (ARC): Tier 1. Fastest speed, stores the “hottest” data.
  • SSD (L2ARC): Tier 2. Holds data recently evicted from ARC that still has a high likelihood of being reread.
  • Pool vdev (HDD/SATA SSD): Base tier. Stores all data, slowest speed.

Keep in mind: L2ARC consumes host RAM for header indexing tables. With the default 8KB volblocksize of a Zvol virtual disk, every 1TB of L2ARC can consume 4GB to 8GB of host RAM. If your server has less than 16GB of RAM, adding a large L2ARC will only degrade performance because it shrinks ARC.

2. SLOG (Separate ZFS Intent Log)

SLOG is not a write cache for all data. ZFS aggregates asynchronous write operations (async writes) into RAM via Transaction Groups (TXGs), then flushes them to disk periodically every 5 seconds.

In contrast, software with strict data integrity requirements—such as PostgreSQL, MySQL, or VM disks mounted over NFS/iSCSI—must use synchronous writes (sync writes). This means the application waits for the disk to acknowledge that data is safely committed to non-volatile storage before proceeding. By default, ZFS writes these records to the on-pool ZIL within the main storage pool, forcing hard drives to continuously read and overwrite concurrently.

Separating the ZIL onto an ultra-fast dedicated SSD is what makes an SLOG setup:

  • Synchronous write records are routed directly to the SLOG with latencies around tens of microseconds.
  • Proxmox immediately replies to the VM: the write is complete.
  • ZFS leisurely bundles the data in RAM and flushes it to the main pool later.

Advanced Deployment: Partitioning and Parameter Tuning

In practice, an SLOG only needs to hold a maximum of 5 seconds worth of sync writes (typically 4GB to 16GB). You can split an enterprise NVMe SSD into two partitions: a small portion for SLOG and the remainder assigned to L2ARC.

1. Accurate Partitioning with sgdisk

# Wipe the existing partition table (WARNING: Erases all data on the disk)
sgdisk --zap-all /dev/disk/by-id/nvme-INTEL_SSDPE2KE016T8_...

# Create partition 1: 16GB for SLOG (plenty for a 10GbE link)
sgdisk -n 1:0:+16G -c 1:"zfs-slog" /dev/disk/by-id/nvme-INTEL_SSDPE2KE016T8_...

# Create partition 2: Remaining capacity for L2ARC
sgdisk -n 2:0:0 -c 2:"zfs-l2arc" /dev/disk/by-id/nvme-INTEL_SSDPE2KE016T8_...

2. Configuring a Mirror for SLOG

Losing an L2ARC drive does not cause data loss; you only lose read caching. But with SLOG, if the drive fails abruptly during a power outage before RAM is flushed to the pool, the last 5 seconds of data will vanish. For production environments, always use a mirror:

zpool add rpool log mirror \
  /dev/disk/by-id/nvme-slog-drive1-part1 \
  /dev/disk/by-id/nvme-slog-drive2-part1

3. Optimizing ZFS Parameters via modprobe

Create a kernel module configuration file to persist cache across reboots and speed up cache population:

cat << 'EOF' > /etc/modprobe.d/zfs.conf
# Set max ARC size (e.g., 16GB: 16 * 1024^3 = 17179869184 bytes)
options zfs zfs_arc_max=17179869184

# Rebuild and preserve L2ARC across host reboots without a cold warm-up
options zfs l2arc_rebuild_enabled=1

# Increase write throttle from RAM to L2ARC to 64MB/s (default 8MB/s is very slow)
options zfs l2arc_write_max=67108864
EOF

# Update initramfs to apply changes
update-initramfs -u -k all

Production Insights & Operational Best Practices

1. Say No to Consumer SSDs for SLOG

Many people deploy a Samsung 980 Pro or Kingston KC3000 as an SLOG and are surprised to see sync write performance plummet drastically. The culprit lies in the cache-flush mechanism. Consumer drives lack power loss protection (PLP), so whenever a sync write command arrives, the controller is forced to flush data from its DRAM buffer directly to the NAND flash chips. This process drives latency up from tens of microseconds to as much as 15–30ms.

An SLOG strictly requires an enterprise SSD equipped with Power Loss Protection (PLP) or Storage Class Memory devices like Intel Optane (P4801X, 900P, M10). These drives feature physical capacitors, allowing instantaneous acknowledgments for on-board DRAM writes with complete safety.

2. Monitoring Hit Rates with arcstat

Don’t guess blindly. After 2–3 days of normal operation, open a terminal to check real-world performance:

arcstat -f time,read,hits,miss,hit%,l2read,l2hits,l2miss,l2hit% 2

Observe the l2hit% column. If the hit rate consistently stays below 5%, your L2ARC is not performing effectively. This typically occurs because your working set is too random/dispersed, or ARC in RAM is already large enough to serve all requests.

3. Safely Removing Cache Drives

Need to swap hardware or modify partitions? Never pull a drive physically before detaching the device from the ZFS pool:

# Remove L2ARC cache (completely safe, can be removed at any time)
zpool remove rpool /dev/disk/by-id/nvme-...-part2

# Remove SLOG (ZFS will automatically flush all pending log records to the pool before disconnecting)
zpool remove rpool /dev/disk/by-id/nvme-...-part1

Once the command completes, the pool returns to its original state without interrupting any running virtual machines.

Share: