Deploying GlusterFS on Ubuntu: A Cost-Effective Distributed Storage Solution – ITFROMZERO

Table of Contents

When Full Disks and System Crashes Loom

About six months ago, I was managing a Web Server cluster for an e-commerce platform. Everything was running smoothly until a Friday night when the system hit red alert: the main server’s disk reached 98% capacity. More frighteningly, it was the sole location for nearly 500GB of product images and user documents.

At that moment, I had two choices. One was to shut down the server to upgrade the drive, accepting at least 30 minutes of downtime. The other was to find a way to mount an external drive, but the legacy code didn’t support distributed storage. After that ‘sweaty’ night, I realized that entrusting data to a single physical hard drive was a fatal mistake.

Why Traditional Storage Is a Fatal Weak Point

If you only use a single Ubuntu Server, you will eventually run into these issues:

Single Point of Failure (SPOF): A drive failure or a motherboard crash means your data is ‘in the hospital.’ Your application will stop working immediately.
Scaling Is a Nightmare: When capacity runs out, buying a larger drive and migrating hundreds of gigabytes of data is extremely time-consuming and prone to errors.
I/O Bottlenecks: A single hard drive has specific read/write limits. When traffic hits 10,000 concurrent users, the drive simply won’t keep up.

GlusterFS: The Sweet Spot Between Cost and Performance

I considered several options:

NFS: Easy to install but still centralized storage. If the NFS server dies, the whole system goes down with it.
Ceph: Very powerful for large Cloud systems. However, Ceph is resource-heavy (needing at least 16-32GB of RAM to run stably) and extremely complex to configure for small teams.
GlusterFS: This was the ‘true love.’ It pools drives from multiple servers into a single partition and supports replication to ensure data is always available.

After testing in a staging environment, I decided to put GlusterFS into production. The results have been impressive: the system has run smoothly for 6 months. Upgrading capacity is now as simple as adding a new node to the cluster without stopping any services.

Steps to Deploy a 3-Node GlusterFS Cluster

I recommend using 3 Nodes instead of 2. This helps avoid ‘Split-brain’ errors (data conflicts when connectivity between nodes is lost). Here is the model I implemented:

srv-01: 192.168.1.10
srv-02: 192.168.1.11
srv-03: 192.168.1.12

Step 1: Configure Node Identification

To allow the servers to ‘call each other by name,’ edit the /etc/hosts file on all 3 machines:

sudo nano /etc/hosts

Add this content to the end of the file:

192.168.1.10 srv-01
192.168.1.11 srv-02
192.168.1.12 srv-03

Step 2: Install GlusterFS

Run the following commands on all 3 Nodes to get the most stable version:

sudo apt update
sudo apt install software-properties-common -y
sudo add-apt-repository ppa:gluster/glusterfs-9 -y
sudo apt update
sudo apt install glusterfs-server -y

Enable the service to start automatically with the system:

sudo systemctl enable --now glusterd

Step 3: Establish the Alliance Between Nodes

On srv-01, type the commands to connect to the other two nodes:

sudo gluster peer probe srv-02
sudo gluster peer probe srv-03

Check the status with: sudo gluster peer status. If you see the status Connected, everything is communicating correctly.

Step 4: Create Bricks (Data Storage Partitions)

In GlusterFS, a ‘Brick’ is simply a storage directory. It is best to use a separate drive formatted with XFS for peak performance. In this example, I am creating a directory on the main drive:

# Run on all 3 nodes
sudo mkdir -p /gluster/data

Step 5: Initialize the Distributed Volume

I will create a Volume named vol_shared with replica 3 mode. This means every file you upload will be automatically copied to all 3 servers. On srv-01, run:

sudo gluster volume create vol_shared replica 3 srv-01:/gluster/data srv-02:/gluster/data srv-03:/gluster/data force
sudo gluster volume start vol_shared

Connecting Clients to the Storage Cluster

To allow your Web Server or App Server to use this storage, install the client package:

sudo apt install glusterfs-client -y

Proceed to mount the network drive:

sudo mkdir /mnt/storage
sudo mount -t glusterfs srv-01:/vol_shared /mnt/storage

The beauty here is: even if you mount via the IP of srv-01, if that machine goes down, the client will automatically look to srv-02 or srv-03 to retrieve data. This process happens in an instant, and your application continues to run normally.

‘Hard-Won’ Lessons After 6 Months of Operation

To keep the system stable, keep these 3 key points in mind:

Internal Network: GlusterFS synchronizes data constantly, so it is bandwidth-intensive. Use at least a 1Gbps LAN. If you use a 100Mbps network, the system will lag when writing large files.
Monitor Capacity: GlusterFS doesn’t perfectly rebalance capacity automatically if one node gets full. Always try to keep your Bricks at similar free space levels.
Auto-mount on Reboot: Don’t forget to add the following line to /etc/fstab to avoid losing the connection after a server restart:
```
srv-01:/vol_shared /mnt/storage glusterfs defaults,_netdev 0 0
```

GlusterFS is an excellent choice if you need a centralized storage system for images, videos, or log files without the cost of expensive SAN solutions. Good luck with your deployment!