The 2 AM Full-Disk Nightmare
At 2:15 AM, my phone started buzzing relentlessly. PagerDuty lit up red: all APIs were down, MySQL had terminated its process, and customers were blowing up support channels. Half-asleep, I SSHed into the VPS, but every command I ran returned the exact same error:
-bash: cannot create temp file for here-document: No space left on device
A quick df -h confirmed the problem: the root partition / was at 100% capacity. Checking /var/log revealed the culprit immediately: access.log had ballooned to 45GB. A vulnerability scan from a botnet had caused a massive spike in requests, and this VPS lacked any automated log rotation or cleanup mechanism.
Seeing an oversized file, many people instinctively run rm -f /var/log/app.log. Don’t do that! The application process still holds the open file descriptor (inode). While ls will no longer show the file, not a single byte of disk space is actually freed. To genuinely reclaim that storage, you would have to restart the service—a major taboo in production environments. To handle this cleanly without service disruptions, Linux provides a powerful built-in utility: logrotate.
How Does logrotate Work?
Logrotate is a utility designed to manage log file lifecycles on Linux. Rather than letting logs grow indefinitely, it intervenes on a scheduled basis:
- Rotate: Renames the active log file (e.g.,
app.logtoapp.log.1) and creates a new, empty file for the application to continue writing to. - Compress: Compresses archived logs into
.gzformat. A 10GB plain text log often shrinks to around 1–1.5GB, saving nearly 90% in disk space. - Remove: Retains a specified number of archives (e.g., 14 days) and automatically purges older ones.
Which Strategy to Choose: copytruncate or postrotate?
Depending on how your application writes logs, you should choose one of two primary approaches:
- Standard mechanism (create + postrotate): Logrotate renames the old log file and immediately creates a new file with the correct permissions. It then sends a POSIX signal (typically
USR1hoặcSIGHUP) telling Nginx or Apache to reopen the log file. This approach optimizes I/O and guarantees zero log loss. - copytruncate mechanism: Used for applications written in Node.js, Python, Go, or Java that lack built-in support for log reload signals. Logrotate copies the contents to a backup file and then truncates the original file in place to 0 bytes. The application keeps its original file descriptor open, eliminating the need for a restart. However, if your server handles 5,000 req/s, logs written during the split second between copying and truncating may be lost.
Practical Production-Ready logrotate Configuration
Logrotate typically runs in the background via cron.daily or a systemd-timer. Global configurations reside in /etc/logrotate.conf. For individual services, you should define separate configuration files under /etc/logrotate.d/ for cleaner management.
Step 1: Create a Configuration File for Your Application
Assume your backend writes logs to /var/log/my-backend/*.log. Open your terminal and create a new file:
sudo nano /etc/logrotate.d/my-backend
Add the following configuration to the file:
/var/log/my-backend/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 0640 appuser appuser
sharedscripts
copytruncate
}
Key directives to understand:
daily: Checks and rotates logs daily. You can also useweeklyormonthly.missingok: Ignores missing log files without raising an error that could disrupt cron.rotate 14: Retains the 14 most recent rotated files. On the 15th day, the oldest archive is automatically deleted.compress: Compresses archived log files with gzip to save storage.delaycompress: Postpones compression of the most recently rotated log until the next cycle. The.1file remains in plain text, making it easy for monitoring scripts to read directly.notifempty: Skips rotation if the log file is empty (0 bytes).create 0640 appuser appuser: Sets specific file permissions and ownership for the newly created log file, securing sensitive log data.copytruncate: Prevents needing to restart the application process during log rotation.
Step 2: Rotate Logs Dynamically by Size
Rotating logs strictly on a daily schedule may not protect you during DDoS attacks or sudden traffic surges. If logs balloon by tens of gigabytes within a few hours, add a size constraint:
/var/log/my-backend/*.log {
daily
maxsize 500M
rotate 10
compress
copytruncate
}
With the maxsize 500M directive, logrotate triggers as soon as the file exceeds 500MB, without having to wait for the daily cycle to finish.
Step 3: Run a Dry Run to Validate Syntax
Over years of managing infrastructure, the single habit that has saved me countless times is always doing a dry run before closing an SSH session. A misplaced curly brace can break the server’s entire daily cron routine.
Use the -d (debug) flag to simulate the entire process without modifying real data:
sudo logrotate -d /etc/logrotate.d/my-backend
If the terminal outputs something similar to the following, your configuration syntax is valid:
reading config file /etc/logrotate.d/my-backend
Allocating hash table for state file, size 64 entries
Handling 1 logs
rotating pattern: /var/log/my-backend/*.log after 1 days (14 rotations)
empty log files are not rotated, old logs are compressed
considering log /var/log/my-backend/app.log
log does not need rotating (log is empty)
Step 4: Perform a Forced Run
Want to verify whether the .gz file generates properly and permissions are set correctly? Use the -f (force) flag to trigger immediate rotation:
sudo logrotate -f /etc/logrotate.d/my-backend
Run ls -la /var/log/my-backend/ to confirm that app.log.1 or app.log.1.gz has been created.
Server Disk Capacity Protection Checklist
Running out of disk space in the middle of the night is an entirely preventable incident. Make logrotate configuration a standard requirement whenever provisioning a new server:
- Always enable
rotatealong withcompressfor both application and web server logs. - Use
copytruncatefor standalone apps, or usepostrotateto send reload signals if supported. - Always run
logrotate -dto verify syntax before disconnecting your session. - Inspect
/var/lib/logrotate/status(or/var/lib/logrotate.statuson older Debian systems) to check exactly when logrotate last executed.
