Storage Concerns When Building a Proxmox Cluster
Anyone building a Proxmox Cluster is likely aiming for High Availability (HA) to keep the system ‘alive’. However, to achieve HA, you must have Shared Storage or Distributed Storage. Otherwise, if a physical node fails, all Virtual Machines (VMs) on it will be ‘stuck’ because the data is trapped on the failed node’s local hard drive.
In a homelab project with 12 VMs and containers, I initially planned to use Ceph due to its popularity. However, Ceph is extremely resource-hungry. It requires at least 3 nodes, and each node consumes about 1GB of RAM for every 1TB of stored data, not to mention the significant CPU overhead. For budget hardware or older servers, running Ceph is like using a sledgehammer to drive a nail—it’s over-engineered and wasteful.
After many nights of tinkering, I switched to LINSTOR. This is a highly streamlined management solution for DRBD (Distributed Replicated Block Device) that delivers impressive performance.
Comparison Table: Why Choose LINSTOR?
Below is a practical comparison to help you see the difference between LINSTOR and its competitors:
| Criteria | Local Storage | Ceph (RBD) | LINSTOR (DRBD) |
|---|---|---|---|
| High Availability (HA) | No | Very High (3+ nodes) | High (only 2 nodes needed) |
| Performance (IOPS) | Native (100%) | Low (~50-70%) | Very High (~90-95% Native) |
| RAM Usage | Near zero | 4GB – 16GB+ | Under 500MB |
| Complexity | Easy | Very Hard | Medium |
How LINSTOR Works
LINSTOR doesn’t store data directly. It acts as a ‘controller’ coordinating DRBD underneath. DRBD replicates data in real-time between disks over the LAN.
- Speed: By operating at the Block-level, data is transmitted as soon as a write command is issued, minimizing latency.
- Efficiency: You only need 2 nodes for HA, instead of the 3-node minimum required by Ceph.
- Flexibility: It works seamlessly with both LVM-Thin and ZFS.
Hardware Requirements
For a smooth deployment, you should have:
- At least 2 Proxmox nodes (I’m using 3 nodes:
pve-01,pve-02,pve-03). - One empty disk on each node (e.g.,
/dev/sdbwith 240GB capacity). - A 1Gbps network is the minimum, but with 10Gbps, LINSTOR will run at ‘blazing speeds’.
Detailed Implementation Steps
1. Repository Installation
LINSTOR is not in the default repositories. You need to add the official LINBIT repo using the following commands on all nodes:
wget -O- https://packages.linbit.com/package-signing-pubkey.asc | gpg --dearmor > /usr/share/keyrings/linbit-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/linbit-keyring.gpg] https://packages.linbit.com/proxmox-8/ proxmox-8 v9" > /etc/apt/sources.list.d/linbit.list
apt update2. Install Controller and Satellite
The system consists of two parts: the Controller (the brain) and the Satellite (the worker on each node).
On the main node (pve-01):
apt install linstor-controller linstor-satellite linstor-client drbd-utilsOn the satellite nodes (pve-02, pve-03):
apt install linstor-satellite drbd-utilsThen, enable the services so they start automatically with the system:
systemctl enable --now linstor-controller linstor-satellite3. Initialize the Cluster and Storage Pool
On the Controller node, connect the nodes into a common network:
linstor node create pve-01 192.168.1.10 --node-type Combined
linstor node create pve-02 192.168.1.11 --node-type Satellite
linstor node create pve-03 192.168.1.12 --node-type SatelliteNext, create the storage area. I chose LVM-Thin because it supports ultra-fast snapshots:
# Create Volume Group from the empty disk
vgcreate drbd_vg /dev/sdb
# Define the pool for LINSTOR management
linstor storage-pool create lvmthin pve-01 storage_data drbd_vg/thin_pool
linstor storage-pool create lvmthin pve-02 storage_data drbd_vg/thin_pool
linstor storage-pool create lvmthin pve-03 storage_data drbd_vg/thin_pool4. Connect to the Proxmox Interface
To allow Proxmox to automatically ‘call’ LINSTOR when creating a VM, you need to install the integration plugin:
apt install linstor-proxmoxAccess the Proxmox Web GUI, go to Datacenter -> Storage -> Add -> LINSTOR. Enter the Controller node’s IP and the pool name storage_data. Don’t forget to create a Resource Group using the following command:
linstor resource-group create proxmox_group --storage-pool storage_data --place 2Pro tip: --place 2 means the data always has 2 replicas. If you want absolute safety across all 3 nodes, change it to 3.
Verification: Live Migration in the Blink of an Eye
Once configured, I tested by installing Ubuntu Server on LINSTOR-Storage. The results were truly impressive. When I clicked Migrate to move the VM from node 1 to node 2, the process took exactly 4.2 seconds.
Throughout the migration, the ping command from an external machine to the VM was never interrupted. Since the data was already synchronized in parallel, Proxmox only needed to transfer the remaining data in RAM. This is the perfect solution for those who need stability without the budget for specialized storage arrays costing thousands of dollars.
Three ‘Critical’ Operational Notes
- Network Bandwidth: Every disk write operation must be pushed over the network. If the network card is congested, the VM will lag immediately. Use at least 2 network cards to separate management traffic from data traffic.
- Split-brain Phenomenon: If two nodes lose connection but keep running, they might overwrite each other’s data. Ensure Quorum is correctly configured on Proxmox to avoid this disaster.
- Kernel Updates: DRBD runs directly in the Linux kernel. Every time you run
apt upgradeand update the kernel, check if the DRBD module has been rebuilt via DKMS.
In summary, LINSTOR is a solid, reliable choice for small and medium systems. It brings the power of distributed storage without requiring the massive hardware resources of Ceph.
