1. A 2:15 AM Incident
The PagerDuty alarm went off relentlessly in the middle of the night. The /var/www/uploads partition on our primary web server hit 98% capacity, leaving less than 4GB of free disk space. This directory held over 180GB of data, consisting of product images, electronic invoices, and customer documents spread across roughly 2.3 million small files.
We needed to urgently migrate this data to our secondary storage cluster (backup-storage-02) before 6:00 AM. Any delay would inevitably cause an I/O bottleneck for morning payment transactions. The on-call engineer from the previous shift had hastily run:
scp -r /var/www/uploads/ [email protected]:/data/backup_uploads/
After nearly two hours of sluggish transfer that pushed barely 60GB, the local network blipped for about two seconds. The scp command dropped immediately with the all-too-familiar error: client_loop: send disconnect: Broken pipe.
The process came to a complete halt. With scp, there is no way to resume an interrupted transfer—you are forced to start over from scratch. Meanwhile, the source disk was edging toward 100%, and the CPU remained pegged at 95% due to default SSH encryption overhead.
2. Why Traditional scp Fails
Looking under the hood of scp, it exposes four fatal flaws when handling large datasets in production:
- No resume support: A connection drop means starting all over. Neither
cpnorscpmaintains transfer state, forcing a full scan and overwrite of every file. - Lack of a Delta Transfer mechanism: Even if a 50GB log file or database dump only changed its last 10MB,
scpstill blindly ships the entire 50GB across the wire. This wastes network throughput and hammers disk I/O. - CPU exhaustion from compressing already compressed files: Millions of JPEG, PNG, or WebP images are already compressed. Enabling stream compression only pegs CPU utilization at 100% with virtually zero reduction in payload size.
- Loss of metadata and permissions: Transferring files without flags to preserve permissions strips read/write access from application users (such as
www-data), triggering widespread HTTP 500 errors after server cutover.
3. Alternatives We Considered
Option 1: Piping tar Directly over SSH
Packaging the stream with tar and streaming it directly through SSH:
tar -cf - /var/www/uploads | ssh [email protected] "tar -xf - -C /data/backup_uploads/"
This bundles millions of small files into a single sequential read stream, significantly cutting down I/O overhead. However, it still suffers from a critical drawback: any network interruption requires restarting from square one. It also cannot handle recurring differential syncs.
Option 2: Using Dedicated LFTP or SFTP Clients
Tools like lftp provide multi-threading and robust resume capabilities. Nonetheless, this approach requires installing extra packages on production and provisioning an SFTP account. In a red-alert production scenario, spending 20 minutes setting up new infrastructure poses unnecessary risk.
Option 3: Using Standard rsync
Rsync thoroughly resolves this problem with its delta-transfer algorithm (rolling checksum). Instead of blindly copying, rsync inspects and transfers only the data blocks that actually changed between source and destination:
rsync -avz /var/www/uploads/ [email protected]:/data/backup_uploads/
This command skips existing files. However, with default flags, rsync can still saturate all available bandwidth or starve the primary application of CPU resources.
4. Battle-Tested rsync Syntax Optimized for Production
To transfer data to the target as fast as possible without degrading the end-user experience, here is the rsync command that rescued our late-night shift:
rsync -avPh \
--delete \
--bwlimit=50000 \
--exclude='*.tmp' --exclude='cache/' \
-e 'ssh -c [email protected] -T -o Compression=no -x' \
/var/www/uploads/ [email protected]:/data/backup_uploads/
Breaking Down Each Parameter:
-a(archive mode): Preserves original file permissions, ownership (user/group), symlinks, and timestamps.-vand-h: Prints verbose progress with human-readable file sizes in MB and GB for easy monitoring.-P(combines--progressand--partial): Essential during unstable network conditions. The--partialflag retains partially transferred files. If the connection drops, rsync resumes right at the interrupted byte rather than deleting and restarting.--delete: Deletes files on the destination if they no longer exist on the source. This ensures a 1:1 mirror and prevents the backup volume from bloating.--bwlimit=50000: Caps bandwidth usage at ~50MB/s (50,000 KiB/s), reserving sufficient network throughput for user traffic.--exclude: Skips thecache/directory and temporary*.tmpjunk files.-e 'ssh ...': Fine-tunes the SSH transport. The[email protected]cipher leverages hardware AES-NI instructions on the CPU. Meanwhile, disabling compression (Compression=no) and X11 forwarding (-x) eliminates needless protocol overhead.
Applying these flags on our Ubuntu 22.04 server (4 vCPUs, 4GB RAM) immediately dropped CPU load from 95% down to 18-22%. All 180GB of data finished syncing safely in just over 55 minutes.
The Trailing Slash Trap
A single trailing slash / in the source path can completely alter the resulting directory structure:
- With a trailing slash on the source (
/var/www/uploads/): Syncs the contents inside the uploads directory directly into the destination directory. - Without a trailing slash on the source (
/var/www/uploads): Rsync creates an additionaluploadssubdirectory inside the destination (resulting in/data/backup_uploads/uploads/).
Before running the actual command, always include the -n (--dry-run) flag to preview planned operations without modifying disk state:
rsync -avPhn --delete /var/www/uploads/ [email protected]:/data/backup_uploads/
Safe Automated Sync Script for Cron Jobs
When running rsync in a background cron job, avoid using -P because it will flood your log files with thousands of progress-bar lines. Instead, pair --partial with flock to prevent overlapping executions:
#!/usr/bin/env bash
set -euo pipefail
LOG_FILE="/var/log/rsync_media.log"
LOCK_FILE="/var/run/rsync_media.lock"
exec 200>"$LOCK_FILE"
flock -n 200 || { echo "[$(date '+%Y-%m-%d %H:%M:%S')] Another rsync process is already running. Aborting." >> "$LOG_FILE"; exit 1; }
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting data synchronization..." >> "$LOG_FILE"
rsync -a --partial \
--delete \
--stats \
--bwlimit=40000 \
--exclude='.git*' \
-e 'ssh -i /root/.ssh/backup_id_rsa -o StrictHostKeyChecking=no' \
/var/www/uploads/ [email protected]:/data/backup_uploads/ >> "$LOG_FILE" 2>&1
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Synchronization complete!" >> "$LOG_FILE"
Understanding rsync flags and the inner workings of SSH handshakes will empower you to manage large-scale data transfers with confidence. No more stressful late-night shifts—your system will always have a rock-solid, secure backup strategy in place.

