A Real-World Story: NAS Disk Hitting 92% and Backups Taking Over 3 Hours
About six months ago, our service cluster running CentOS Stream 9 started hitting a bottleneck during backups. The cluster consisted of two Node.js/PHP web applications and a MySQL cluster hosting around 40GB of transactional data.
Every night, a cron job ran a script to dump MySQL, archived the source code directories using tar, and transferred them to our NAS via rsync. After four months of operation, three critical bottlenecks emerged:
- The 2TB drive on the NAS reached a critical 92% capacity, even though the retention policy kept only the last 14 days.
- Backup duration skyrocketed from 45 minutes to 3 hours and 20 minutes. High disk I/O caused application queries to stall (I/O wait spiked past 35%).
- Data transferred across the local network was entirely plaintext, lacking end-to-end encryption.
Why Traditional rsync and tar Fall Short
The legacy approach seemed straightforward on the surface, but it suffered from three major technical hurdles underneath:
1. Lack of Chunk-Level Deduplication
In a 40GB database file, only about 2–3% of records change daily. Even so, mysqldump or tar still generates a completely new file with a distinct checksum. While rsync optimizes bandwidth during transfer, you still end up storing 14 independent full copies on the NAS server to maintain daily rollback history. Roughly calculated, 14 copies of the database alone consumed 560GB.
2. Hardlink Mechanisms Are Helpless Against Database Dumps and Compressed Archives
Many engineers use rsync’s --link-dest flag to save disk space. This works smoothly for static images or rarely modified media files. However, it is utterly useless for .tar.gz compressed archives or SQL dumps. As byte offsets shift and inodes change, rsync is forced to store the entire new file.
3. Gzip Saturates a Single CPU Core with Subpar Compression Ratios
Traditional gzip processes peg a single CPU core at 100% for hours on end, consuming server resources while slowing down downstream workflows.
Evaluating Three Options Before Implementation
Before scrapping the old script, I set up a staging environment to benchmark three potential solutions:
- Option 1: Rsync + ZFS/Btrfs Snapshots (on the NAS). Snapshot creation is lightning fast. The downside is requiring a reformat of the NAS drives, consuming extra RAM for the ZFS pool, and failing to handle encryption directly on the CentOS Stream 9 client.
- Option 2: Restic. Built in Go, it offers excellent cloud backend support. However, once the repository surpassed 1.5TB in our testbed, maintenance commands like
restic checkandpruneconsumed over 3.2GB of RAM, easily causing memory exhaustion on smaller VPS instances. - Option 3: BorgBackup (Borg). Borg splits data into dynamic chunks via rolling hashes, storing only unique chunks in the repository. Furthermore, Borg supports high-speed Zstandard (zstd) compression, client-side AES-256 encryption, and mounting backup archives as folders via FUSE.
Step-by-Step Guide to Installing and Standardizing BorgBackup on CentOS Stream 9
BorgBackup checked all three boxes: space efficiency, high speed, and low RAM footprint. Below are the steps implemented in our production environment.
Step 1: Install BorgBackup and FUSE from EPEL
The packages are readily available in the EPEL (Extra Packages for Enterprise Linux) repository. Install Borg alongside the FUSE module to enable archive mounting later:
# Enable EPEL and install Borg along with FUSE
sudo dnf install -y epel-release
sudo dnf install -y borgbackup fuse
Verify the installed version:
borg --version
# Sample output: borg 1.2.8
Step 2: Initialize the Repository and Safely Store the Repokey
In this example, the backup target directory resides at /mnt/backup-storage/borg-repo (mounted from the NAS via NFS). The recommended encryption mode is repokey-blake2: the encryption key is stored in the repository header and protected by a passphrase.
# Create the repository directory
sudo mkdir -p /mnt/backup-storage/borg-repo
# Initialize the repository
sudo borg init --encryption=repokey-blake2 /mnt/backup-storage/borg-repo
You will be prompted to enter a passphrase. Choose a strong passphrase and save it in a password manager; losing the passphrase means losing all your data. Next, export a backup key to a secure file:
# Export the backup key to a text file and store it in an isolated location
sudo borg key export /mnt/backup-storage/borg-repo /root/borg-backup-repo.key
Step 3: Run the Initial Backup Using ZSTD Compression
The backup targets include the web source code at /var/www and SQL dumps at /var/backups/db. The zstd,3 algorithm delivers a compression ratio comparable to gzip level 6 while running nearly 3 times faster:
# Set a temporary passphrase for the session
export BORG_PASSPHRASE='YourSecurePassword123@'
# Create the first archive
sudo -E borg create --stats --progress --compression zstd,3 \
/mnt/backup-storage/borg-repo::'{now:%Y-%m-%d_%H:%M}' \
/var/www \
/var/backups/db \
--exclude '*.log' \
--exclude '/var/www/*/cache'
Once completed, the command prints a storage summary table. By the second day, you will notice a drastic reduction in the Deduplicated size column. The actual volume written to disk drops to a few hundred megabytes instead of tens of gigabytes.
Step 4: Automate with a Shell Script and Systemd Timer
Instead of a basic cron job, create a script that combines all three tasks: creating archives, pruning old backups, and reclaiming disk space (compacting). Create the file /usr/local/bin/borg-backup.sh:
#!/usr/bin/env bash
set -euo pipefail
export BORG_REPO="/mnt/backup-storage/borg-repo"
export BORG_PASSPHRASE="YourSecurePassword123@"
ARCHIVE_NAME="$(hostname)-$(date +%Y-%m-%d_%H%M%S)"
echo "[$(date)] Starting Borg backup process..."
# 1. Create a new archive
borg create --stats --compression zstd,3 \
::"$ARCHIVE_NAME" \
/var/www \
/etc \
/var/backups/db \
--exclude '*/cache' \
--exclude '*/tmp'
echo "[$(date)] Archive complete. Starting cleanup of old backups..."
# 2. Prune old archives: keep last 7 days, 4 weeks, 6 months
borg prune -v --list \
--keep-daily=7 \
--keep-weekly=4 \
--keep-monthly=6
# 3. Reclaim free disk space after deleting metadata
borg compact
echo "[$(date)] Entire backup process finished successfully!"
Restrict permissions so that only the root user can read and execute the script:
sudo chmod 700 /usr/local/bin/borg-backup.sh
On CentOS Stream 9, a Systemd Timer replaces cron effectively thanks to fine-grained I/O priority management and centralized logging via journalctl.
Create the service unit file /etc/systemd/system/borg-backup.service:
[Unit]
Description=BorgBackup Daily Service
After=network.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/borg-backup.sh
Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7
Create the timer file /etc/systemd/system/borg-backup.timer to trigger at 2:30 AM:
[Unit]
Description=Trigger BorgBackup daily at 02:30 AM
[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
[Install]
WantedBy=timers.target
Reload the systemd configuration and enable the timer:
sudo systemctl daemon-reload
sudo systemctl enable --now borg-backup.timer
# Check the next scheduled execution time
sudo systemctl list-timers borg-backup.timer
Step 5: Restore and Verify Data
A backup is only as good as its restore test. With Borg, you have two flexible methods to recover files depending on the scenario:
Method 1: Extract the entire archive directly
export BORG_PASSPHRASE="YourSecurePassword123@"
# List archives in the repo
sudo -E borg list /mnt/backup-storage/borg-repo
# Extract data from a specific backup archive into a temporary restore directory
mkdir -p /tmp/restore && cd /tmp/restore
sudo -E borg extract /mnt/backup-storage/borg-repo::centos9-prod-2026-10-20_023000
Method 2: Mount the repository as a filesystem to retrieve individual files
When developers only need to recover a single configuration file or an old SQL table, you don’t need to unpack the entire large archive. Simply mount the repo via FUSE and browse files using standard commands:
sudo mkdir -p /mnt/borg-mount
# Mount the repo as read-only
sudo -E borg mount /mnt/backup-storage/borg-repo /mnt/borg-mount
# Browse files just like a regular partition
ls -la /mnt/borg-mount
cp /mnt/borg-mount/centos9-prod-2026-10-20_023000/var/www/html/app/config.php /tmp/
# Once finished, unmount the directory
sudo umount /mnt/borg-mount
Results After 6 Months in Production
Real-world metrics gathered from our CentOS Stream 9 cluster clearly demonstrated the impact:
- Storage footprint: The 14-day backup retention previously consumed approximately 1.8TB on the NAS. Currently, the Borg repository consolidates all this data into just 310GB of disk space, saving nearly 83% of storage.
- Execution time: Total nightly backup time plummeted from 3 hours and 20 minutes down to just 6–8 minutes, as unchanged chunks are neither re-read nor sent over the network.
- System load: Thanks to
Nice=19and a lowered I/O priority class, the server experienced zero latency spikes or query lag for end users at 2:30 AM.

