Data Storage: When One Hard Drive Is Not Enough
If you’ve ever stayed up all night because a server’s hard drive suddenly died, you know the horror of watching customer data vanish in an instant. I used to use rsync for periodic backups, but there’s always latency. If the server crashes at 2 AM, the 15 minutes of precious data lost between backups is still a disaster for the business. To achieve High Availability, we need a “harder” solution: GlusterFS.
GlusterFS is a distributed file system that allows you to aggregate multiple hard drives from different physical servers into a single giant storage pool. Its real-time replication capability is its most valuable feature. When one node fails, the other continues to operate normally. End users won’t even notice any disruption.
After migrating my systems from the aging CentOS 7 to CentOS Stream 9, I realized that setting it up on the new platform involves some important changes regarding repositories. If you’re not aware of these, you might get stuck with dependency errors.
Preparing the Lab Environment
In this guide, I’ll be using 2 CentOS Stream 9 nodes. In a production environment, I recommend using at least 3 nodes. Three nodes help the system achieve Quorum, avoiding *split-brain*—a common error where two nodes cannot agree on which data is the most recent.
- Node 1: gfs-node01 (192.168.1.101)
- Node 2: gfs-node02 (192.168.1.102)
- Secondary Hard Drive: Each node is equipped with an additional 20GB drive (usually
/dev/sdb). A small note: Never use the/(root) partition to run GlusterFS. If the drive fills up, the entire operating system will freeze immediately.
Declaring the Hosts File
Nodes need to recognize each other via hostname instead of just raw IP addresses. Edit the /etc/hosts file on both nodes:
192.168.1.101 gfs-node01
192.168.1.102 gfs-node02Installing GlusterFS on CentOS Stream 9
The default CentOS 9 repository does not include GlusterFS. You need to enable the repo from the SIG (Special Interest Group) to get the most stable build.
1. Enabling the Repository
Run the following commands on both nodes:
# Install the release package to get the gluster repo
sudo dnf install centos-release-gluster -y
# Enable the CRB (Code Ready Builder) repo - replacement for the old PowerTools
sudo dnf config-manager --set-enabled crb2. Deploying the Service
sudo dnf install glusterfs-server -y
# Start and enable the service to run at boot
sudo systemctl enable --now glusterd
# Quickly check the status
sudo systemctl status glusterd3. Configuring the Firewall
Many people struggle to understand why nodes can’t connect even though the service is running. GlusterFS uses port 24007 and a range of ports starting from 49152 for each Brick. The fastest and safest way for an internal network is:
sudo firewall-cmd --add-service=glusterfs --permanent
sudo firewall-cmd --reloadSetting Up the Cluster and Volume
Connecting the Nodes (Peering)
From gfs-node01, execute the command to connect to its neighbor:
sudo gluster peer probe gfs-node02To ensure both have successfully “shaken hands”, type:
sudo gluster peer statusWhen you see the line Peer in Cluster (Connected), you’re halfway there.
Creating a Storage Brick
Here, I’m formatting the /dev/sdb drive with the XFS format to optimize performance for GlusterFS:
sudo mkfs.xfs /dev/sdb
sudo mkdir -p /data/glusterfs/myvol/brick1
sudo mount /dev/sdb /data/glusterfs/myvol/brick1
# Remember to add to /etc/fstab to auto-mount the drive after rebooting.Creating a Replicated Volume
Now it’s time to create a volume named gv0 in replica 2 mode (1:1 copy to both nodes):
sudo gluster volume create gv0 replica 2 gfs-node01:/data/glusterfs/myvol/brick1/brick gfs-node02:/data/glusterfs/myvol/brick1/brick force
sudo gluster volume start gv0Testing and Operation
To check if the Volume is “healthy”, use the command:
sudo gluster volume infoIf you want a closer look at latency and running processes, try:
sudo gluster volume status gv0Connecting from the Client
On the client machine, you just need to install glusterfs-client and mount it simply as follows:
sudo mount -t glusterfs gfs-node01:/gv0 /mnt/storageA small tip: Even if you mount via node01’s IP, if this node dies, the client will automatically redirect to node02. The metadata downloaded during mount allows the client to know the structure of the entire cluster.
Real-world Experience: Don’t Fall into Split-brain
Split-brain occurs when the connection between 2 nodes is suddenly cut while both are writing data. GlusterFS will lock that file because it doesn’t know which version is the “official” one.
- Best solution: Always use 3 nodes or add an Arbiter node (a witness node that doesn’t consume much storage space).
- How to handle it: Check for faulty files using the command
gluster volume heal gv0 info split-brain. You will have to manually choose which node has the correct data to restore.
Building a distributed storage system isn’t hard if you follow the correct sequence. With GlusterFS on CentOS Stream 9, I can finally sleep soundly knowing my data always has a safe backup copy. Good luck with your deployment!
