Early this year, our internal media storage cluster hit its capacity limit. Our team was uploading roughly 120,000 images and PDF files daily, generating over 300GB of new data. We faced a demanding challenge: immediately replicate all files from the primary node to a secondary node for hot standby backup and read-load balancing across Nginx.
Deploying Ceph or GlusterFS at this scale would have been massive overkill. With only two nodes, a distributed cluster would be overly complex and waste significant RAM on heartbeat maintenance. After weighing our options, our team went with the classic combo: Lsyncd + Rsync on CentOS Stream 9. Over six months in production, this setup has consumed under 40MB of RAM and required virtually zero maintenance.
Quick Start: Up and Running in 5 Minutes
The deployment topology consists of two servers on the same 10Gbps local network:
- Source Server (Source Node):
192.168.1.10 - Target Server (Target Node):
192.168.1.20 - Sync directory:
/var/www/uploads/
Step 1: Set Up Passwordless SSH Keys
Lsyncd runs in the background as root and invokes Rsync over SSH. Therefore, the Source server needs direct SSH access to the Target server without password prompts.
On the Source Server, generate an Ed25519 key pair and copy it to the Target:
ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519
ssh-copy-id -i ~/.ssh/id_ed25519.pub [email protected]Verify the setup by running ssh [email protected]. If you get straight into the target shell without entering a password, the first step is complete.
Step 2: Install Rsync and Lsyncd
Rsync must be installed on both nodes, whereas Lsyncd is only needed on the Source. On CentOS Stream 9, the package is readily available in the EPEL repository.
Run the following commands on both servers:
sudo dnf install -y epel-release
sudo dnf install -y rsyncNext, install Lsyncd on the Source Server:
sudo dnf install -y lsyncdStep 3: Create the Lsyncd Configuration File
Lsyncd configurations use Lua syntax. Back up the default file and create a new one at /etc/lsyncd.conf:
sudo cp /etc/lsyncd.conf /etc/lsyncd.conf.bak
sudo nano /etc/lsyncd.confStandard configuration for a local network:
settings {
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd.status",
statusInterval = 10,
maxProcesses = 2
}
sync {
default.rsyncssh,
source = "/var/www/uploads",
host = "[email protected]",
targetdir = "/var/www/uploads",
delay = 3,
rsync = {
archive = true,
compress = false,
_extra = { "--bwlimit=60000" }
}
}Step 4: Start and Verify
Ensure the /var/www/uploads directory exists on both servers. Then enable and start the service on the Source server:
sudo systemctl enable --now lsyncd
sudo systemctl status lsyncdCreate a quick test file: touch /var/www/uploads/demo-sync.txt. Check the uploads directory on the Target node; the file should appear within 3 to 5 seconds.
Under the Hood: How Lsyncd Works
A common question is: why not just run an rsync cron job once every minute?
The bottleneck comes down to disk I/O. When a directory swells to 800,000 files, every rsync invocation must traverse the entire directory tree to compare timestamps and file sizes. Disk I/O thrashes continuously and CPU usage spikes, even if only two new files were actually created.
Lsyncd solves this problem cleanly by combining two processing layers:
- Linux inotify: Lsyncd listens directly for Linux kernel filesystem events (create, write, delete, move). The kernel notifies it immediately of modified files, eliminating the need to scan disks.
- Batching mechanism (Aggregator): Lsyncd does not invoke rsync the instant an event occurs. Instead, it aggregates changes within a delay window (configured above as
delay = 3seconds). Once the 3-second window elapses, it bundles the changed files and invokes rsync to push only that specific batch in a single run.
Key Configuration Parameters Explained
- default.rsyncssh: Lsyncd leverages SSH to transfer files. This is secure and requires no additional exposed service ports.
- delay = 3: The event aggregation delay in seconds. Never set this to 0. If a user is uploading a 500MB video, setting delay = 0 causes rsync to trigger prematurely while the file is still being written, wasting resources and causing file corruption.
- compress = false: On local networks of 1Gbps or higher, disable gzip compression. The CPU avoids wasting compression/decompression cycles, resulting in noticeably faster throughput. Only enable compression when syncing across slower internet connections under 50Mbps.
- –bwlimit=60000: Limits bandwidth to roughly 60MB/s (rsync units are in KiB/s). This flag prevents rsync from saturating LAN bandwidth needed by adjacent services like databases.
Production Tuning for Large Filesystems
Default Linux kernel limits are only suited for development environments. When directory trees grow beyond hundreds of thousands of files, systems will run into issues without proper kernel tuning.
1. Increase inotify Watch Limits
By default, CentOS Stream 9 allocates only 8,192 inotify watch descriptors. Once the total number of directories exceeds this limit, Lsyncd immediately crashes with the infamous error: No space left on device, even when hundreds of gigabytes remain free on disk.
Check the current limits:
cat /proc/sys/fs/inotify/max_user_watchesRaise the limit to 1 million directories using a sysctl configuration file:
cat << 'EOF' | sudo tee /etc/sysctl.d/99-lsyncd.conf
fs.inotify.max_user_watches = 1048576
fs.inotify.max_user_instances = 1024
fs.inotify.max_queued_events = 32768
EOF
sudo sysctl --systemEach watch consumes approximately 1KB of kernel memory on 64-bit systems. Tracking 1 million directories uses at most around 1GB of RAM—a worthwhile trade-off for rock-solid system stability.
2. Switch to Rsync Daemon for High Performance
Syncing over SSH is secure but introduces CPU overhead due to encryption. If both nodes reside within a trusted private subnet and need to sync tens of thousands of small files hourly, running a native rsync daemon over port 873 reduces CPU load by roughly 35–40%.
On the Target Server, edit /etc/rsyncd.conf:
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log
[media_mirror]
path = /var/www/uploads
hosts allow = 192.168.1.10
read only = false
uid = nginx
gid = nginxOpen the port in firewalld and start the rsync daemon:
sudo firewall-cmd --permanent --add-service=rsyncd
sudo firewall-cmd --reload
sudo systemctl enable --now rsyncdOn the Source Server, update the sync block in /etc/lsyncd.conf:
sync {
default.rsync,
source = "/var/www/uploads",
target = "192.168.1.20::media_mirror",
delay = 3,
rsync = {
archive = true,
compress = false
}
}3. Exclude Junk and Temporary Files
Avoid wasting I/O on log files or temporary files generated by text editors. Add an exclude filter to your configuration:
sync {
default.rsyncssh,
source = "/var/www/uploads",
host = "[email protected]",
targetdir = "/var/www/uploads",
exclude = {
'*.tmp',
'*.swp',
'.git*',
'cache/**'
}
}Hard-Earned Production Lessons
A setup running smoothly in the lab does not guarantee peace in production. Here are three real-world lessons our team learned the hard way through actual downtime:
Exercise Extreme Caution with the delete Parameter
By default, default.rsyncssh enables file deletion mirroring. That means deleting a file on the Source automatically deletes it on the Target as well.
This turns a hot backup into a double-edged sword: an accidental rm -rf on the Source wipes out data on the Target within 3 seconds. If your primary goal is safe data redundancy, disable automatic deletion entirely:
delete = falseIf you still need to purge orphaned files to reclaim disk space, a safer alternative is delete = 'running'. This parameter ensures Lsyncd deletes files on the target only after completing the synchronization of all new files, preventing data loss during network hiccups.
Lsyncd Is Strictly a One-Way Solution (Master-Slave)
Never set up bidirectional cross-sync (node A pushing to node B while node B pushes back to node A). Lsyncd has no conflict resolution mechanism. When files are modified concurrently on both ends, the nodes trigger endless event loops and race conditions. This spikes CPU usage to 100%, generates tens of gigabytes of logs, and crashes the service. If your business requirements mandate active-active writes on both ends, use GlusterFS, MooseFS, or MinIO Object Storage instead.
Auto-Recover from Kernel OOM Kills
When resource-heavy applications like PHP-FPM or Elasticsearch exhaust system RAM, the Linux OOM Killer may terminate lsyncd to free up memory. To prevent the service from failing silently, create a systemd override:
sudo systemctl edit lsyncdAdd the following configuration to make systemd automatically restart the process after 5 seconds:
[Service]
Restart=always
RestartSec=5sTo monitor synchronization health, inspect the status file:
cat /var/log/lsyncd/lsyncd.statusIf Inotify watches matches your actual directory count and Delay queues consistently returns to 0, your system is running reliably.
