How to Install and Configure Built-in Ceph Storage on Proxmox VE: Self-Healing Distributed Storage - ITFROMZERO - Share tobe shared! A hands-on guide to deploying Ceph Storage on a 3-node Proxmox VE cluster. Covers dedicated 10GbE networking, PLP enterprise SSD optimization, and failover validation during node outages.

Table of Contents

Introduction: The Pains of Centralized Storage and the HA Dilemma

Historically, storage for Proxmox clusters has been a major headache for sysadmins, leaving them with only two practical paths: local storage (Local ZFS/LVM) or dedicated external NAS/SAN setups (such as TrueNAS over iSCSI/NFS). Both options come with frustrating bottlenecks.

Relying on local drives traps virtual machines on their host when it crashes, rendering High Availability (HA) virtually useless—unless you set up Proxmox VE replication with ZFS. Conversely, investing in an external SAN/NAS appliance for shared storage introduces a critical single point of failure (SPOF). A single power supply failure on the NAS or a faulty storage switch will bring down all 10–20 VMs across your nodes simultaneously.

About six months ago, I decided to migrate my entire homelab infrastructure to a Hyperconverged Infrastructure (HCI) model using built-in Ceph on Proxmox VE 8. The lab consists of a 3-node mini server cluster hosting 12 VMs and 8 LXC containers—ranging from PostgreSQL clusters and Redis to GitLab. After six months of stress testing—hot-unplugging network cables and pulling SSDs under heavy write workloads—Ceph proved its remarkable resilience through self-healing capabilities without requiring expensive enterprise SAN hardware.

Core Concepts of Ceph on Proxmox VE

Ceph is an open-source distributed storage system. It aggregates physical drives across all cluster nodes into a unified storage pool. Data written to Ceph is split into 4MB objects, which the CRUSH algorithm then deterministically distributes across servers without relying on a centralized metadata server for lookups.

To smoothly operate a Ceph cluster on Proxmox, you only need to master four essential components:

OSD (Object Storage Daemon): A daemon mapped 1:1 to each physical drive (NVMe/SSD). The OSD is responsible for reading, writing, replicating, and monitoring the health of that specific drive.
MON (Ceph Monitor): The master brain maintaining the cluster state map (Cluster Map). To maintain quorum voting, you must have an odd number of MONs (a minimum of 3 MONs across 3 separate nodes).
MGR (Ceph Manager): Collects performance metrics (IOPS, latency, capacity) and provides an API for the Proxmox web UI to display real-time graphs.
RBD (RADOS Block Device): The block device format that Proxmox uses to provision virtual disks (raw format) for KVM virtual machines, supporting instant snapshots and seamless thin cloning.

The self-healing mechanism operates based on replication parameters: size/min_size. When set to size=3 and min_size=2, Ceph maintains three replicated copies across the three nodes. If node 3 suddenly loses power, Ceph detects that only two copies remain. Because it still meets the min_size=2 threshold, VMs running on nodes 1 and 2 continue to read and write normally without interruption or latency spikes.

Hardware Requirements: Hard-Earned Lessons

Ceph is powerful, but that power comes with strict hardware demands. Avoid running Ceph in production unless your infrastructure satisfies these three requirements:

Node Count: A minimum of three distinct physical nodes. Avoid two-node clusters unless you want to invite split-brain disasters during network blips.
Network Infrastructure: Strictly avoid 1Gbps networks. When a node drops and rejoins, Ceph initiates a rebalance process. A 1Gbps link will saturate immediately, driving I/O latency from 1ms to over 1000ms and freezing your VMs. The bare minimum is 2.5Gbps; the ideal setup is 10Gbps SFP+ using Direct Attach Copper (DAC) cables for sub-0.1ms latency.
SSD Storage: Enterprise SSDs with PLP (Power Loss Protection)—such as the Intel D3-S4510, Micron 5300 PRO, or Kioxia CD6—are mandatory. Consumer SSDs (like the Samsung 980 or Crucial P3) lack write-cache protection capacitors, forcing Ceph BlueStore to continuously flush data. As a result, random write speeds plummet drastically from 400MB/s down to a crippling 10–25MB/s.

Step-by-Step Walkthrough: Installing and Configuring a 3-Node Ceph Cluster

Step 1: Configure a Dedicated Network for Ceph

We will separate the PVE management IP subnet from the internal Ceph cluster interconnect subnet. Strictly avoid letting Ceph cluster traffic compete with regular VM traffic for bandwidth, much like isolating a Corosync redundant network to prevent cluster disruption.

Open /etc/network/interfaces on each node (Node 1 – pve-01 in this example):

# Standard PVE management and VM network (1Gbps)
auto vmbr0
iface vmbr0 inet static
    address 192.168.1.11/24
    gateway 192.168.1.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0

# Dedicated 10GbE network for the Ceph cluster
auto enp3s0
iface enp3s0 inet static
    address 10.10.10.11/24
    mtu 9000

Repeat this configuration for Node 2 (10.10.10.12/24) and Node 3 (10.10.10.13/24). Apply changes using ifreload -a. Setting mtu 9000 (Jumbo Frames) enables the NIC to handle larger payload frames, reducing CPU interrupt overhead by roughly 15% during high-throughput bursts, a key technique when optimizing high-speed networking on Linux hypervisors.

Step 2: Install Ceph Reef Across All Nodes

Run the installation command for Ceph Reef (the default stable release on Proxmox VE 8) on all three servers via SSH:

pveceph install --version reef

Step 3: Initialize the Ceph Cluster on the First Node

Execute the initialization command only once on Node 1 (pve-01), specifying the 10G subnet configured earlier:

pveceph init --network 10.10.10.0/24

This command generates the default configuration file /etc/pve/ceph.conf and automatically replicates it to the other two nodes via the Proxmox Cluster File System (pmxcfs).

Step 4: Create Monitors and Managers on All Three Nodes

For maximum fault tolerance, deploy MON and MGR daemons on every node:

# Run sequentially on pve-01, pve-02, and pve-03:
pveceph mon create
pveceph mgr create

Verify quorum status by running:

ceph mon stat

Output showing quorum 0,1,2 pve-01,pve-02,pve-03 confirms that all three nodes have successfully established quorum.

Step 5: Initialize OSDs from Raw Drives

Assuming each node has an unused NVMe drive at /dev/nvme0n1, wipe any leftover metadata or partitions first to prevent initialization conflicts:

# Run on each node before provisioning the OSD
wipefs -a /dev/nvme0n1
pveceph osd create /dev/nvme0n1

Once completed across all three nodes, check the OSD distribution tree with:

ceph osd tree

All OSDs should show a status of up and in with their storage capacities correctly reported.

Step 6: Create the Ceph Pool and Attach Storage for VMs

Next, create a storage pool named ceph-vm-pool with 3-way replication:

pveceph pool create ceph-vm-pool --size 3 --min_size 2 --add_storages 1

The --add_storages 1 flag automatically registers this pool in /etc/pve/storage.cfg. When provisioning a new VM, simply select ceph-vm-pool under the Hard Disk settings. The VM’s virtual disk data will then be automatically striped and distributed across all three physical hosts, enabling seamless zero-downtime live migration across the cluster.

Step 7: Real-World Failover Simulation

Validate self-healing with a realistic stress test: boot a Linux VM on Node 1 and execute a continuous write benchmark:

fio --name=test --ioengine=libaio --rw=randwrite --bs=4k --size=2G --numjobs=1 --iodepth=16 --runtime=60 --time_based

While the VM is actively writing data at around 180MB/s, pull the power cord on Node 3. Then return to Node 1 and check cluster status:

ceph status

The terminal will report a warning state (HEALTH_WARN):

  cluster:
    id:     a7b28c31-7e89-4d22-b91c-123456789abc
    health: HEALTH_WARN
            1/3 mons down, quorum pve-01,pve-02
            1 host down
            1/3 osds down

  services:
    mon: 3 daemons, 2 in quorum (pve-01, pve-02)
    mgr: pve-01(active), standbys: pve-02
    osd: 3 osds: 2 up, 2 in

The fio workload inside the VM continues running without throwing any I/O errors. Latency blips slightly from 1.2ms to 2.5ms for the first three seconds before stabilizing (though you can also limit disk I/O on Proxmox to keep test benchmarks from impacting other guests).

If Node 3 remains offline for more than 10 minutes (governed by the default mon_osd_down_out_interval of 600 seconds), Ceph will automatically mark its OSDs as out. If spare disks are present in the cluster, Ceph immediately replicates degraded placement groups to healthy drives to restore the full three-copy redundancy.

Conclusion

Built-in Ceph on Proxmox VE offers a compelling solution for high-performance distributed storage without bloating your budget on bulky enterprise SAN systems. Before rolling this out to production, keep these three golden rules in mind:

Maintain an odd number of nodes (at least three) to ensure reliable quorum voting.
Invest in a dedicated 10Gbps network with DAC or fiber optics to eliminate replication and sync latency.
Use enterprise-grade SSDs/NVMes with PLP to safeguard sync-write throughput under BlueStore.

Check all three boxes, and you’ll have an enterprise-ready virtualization cluster: resilient, easily scalable, and capable of healing itself through hardware failures without waking you up in the middle of the night.