Log Rotation on Linux with Logrotate: Rescuing Your Disk Drive in the Middle of the Night

Context & Why It’s Needed: When Logs Crash Your Server at 2 AM

Have you ever received a 2 AM call reporting a system error, unable to write data because the hard drive was full? I’ve experienced that feeling. On production servers, especially the VPS instances I’ve managed for the past 3 years, log files can swell rapidly. These are valuable records of the system and applications, helping me monitor activities, find errors, and understand events.

However, if not managed carefully, these log files can quickly consume all disk space. I recall a server running WordPress and Nginx that continuously reported 502 errors. Upon checking, the Nginx log file had grown to several tens of GBs, and syslog was similar.

The consequence was a full disk, preventing the system from writing any more data, including sessions or cache, leading to extended downtime. Having managed over 10 Linux VPS instances in 3 years, I’ve concluded that every change needs thorough review before applying it to a production environment – especially minor details like logs. **Logrotate** is the solution to this problem.

Logrotate is a utility designed to automatically rotate, compress, remove, and mail log files. It helps prevent system overload due to old and excessively large log files, while still preserving the necessary log history for inspection. Without Logrotate, you might find yourself awake all night again due to similar incidents.

Installing Logrotate on Linux

Typically, most popular Linux distributions come with logrotate pre-installed. If your server doesn’t have it, installation is straightforward:

On Debian/Ubuntu:

sudo apt update
sudo apt install logrotate

On CentOS/RHEL/Fedora:

sudo yum install logrotate
# Or with newer versions
sudo dnf install logrotate

After installation, logrotate is typically activated automatically via a cron job or systemd timer. You can verify its periodic operation by checking configuration files in /etc/cron.daily/ or /etc/systemd/system/timers.target.wants/ (for systemd).

Detailed Logrotate Configuration

Logrotate uses two main types of configuration files:

• /etc/logrotate.conf: The main configuration file, containing default settings and references to other individual configuration files.
• /etc/logrotate.d/: A directory containing separate configuration files for each application or service. This is the ideal place to create custom configuration files, for example, for Nginx, Apache, MySQL, or Python applications.

Main Configuration File: /etc/logrotate.conf

When you open this file, you will see the default configuration lines:

sudo cat /etc/logrotate.conf

# Logrotate.conf is the default configuration file for the entire system.
# Refer to man logrotate for more details.

monthly             # Rotate logs monthly
rotate 4            # Keep 4 old rotated logs
create              # Create new log file after rotation
include /etc/logrotate.d  # Include configuration files in this directory

/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0600 root utmp
    rotate 1
}

# Others...

Below is an explanation of some important directives:

• daily, weekly, monthly, yearly: Defines the log rotation frequency. Choose the option that suits your needs.
• rotate N: Specifies the number of rotated log files to keep. For example, rotate 4 will keep the 4 oldest versions.
• create [mode owner group]: Creates a new empty log file with specified permissions (mode), owner, and group after the old file is rotated. If not specified, the original file’s permissions will be retained.
• include /path/to/directory: Instructs logrotate to read additional configuration files from the specified directory.

Application-Specific Configuration: /etc/logrotate.d/

This is where you customize logrotate for specific applications. Each file in this directory defines how to rotate logs for one or more files. For example, we will create a configuration file for Nginx:

sudo nano /etc/logrotate.d/nginx

/var/log/nginx/*.log {
    daily             # Rotate logs daily
    missingok         # Don't report an error if the log file doesn't exist
    rotate 7          # Keep 7 old log files (equivalent to 1 week)
    compress          # Compress old log files to save space
    delaycompress     # Delay compression of the most recent log file until the next cycle
    notifempty        # Don't rotate logs if the file is empty
    create 0640 www-data adm # Create new log file with permissions 0640, owner www-data, group adm
    sharedscripts     # Only run postrotate/prerotate scripts once for all configured files
    postrotate        # Start script to run after log rotation
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 `cat /var/run/nginx.pid`
        fi
    endscript         # End postrotate script
}

Further explanation of important directives:

• /var/log/nginx/*.log: The path to the log files that need rotation. Wildcard characters like * can be used.
• missingok: If the log file does not exist, logrotate will skip it without reporting an error. This is useful for applications that don’t always create logs.
• compress: Compresses rotated log files using gzip (default), significantly saving disk space. For example: access.log.1 will become access.log.1.gz.
• delaycompress: When using compress, the most recent log file (ending with .1) will not be compressed immediately but will wait until the next rotation cycle. This is beneficial if an application is still reading the .1 file after rotation.
• notifempty: Prevents logrotate from rotating empty log files.
• postrotate / endscript: Commands within this block will execute after logrotate has completed the file rotation. For Nginx, a USR1 signal needs to be sent for Nginx to reopen new log files after they have been rotated. Otherwise, Nginx will continue writing to the old, renamed log file.
• prerotate / endscript: Similar to postrotate, but these commands will run before log rotation.
• dateext: Adds the date to the rotated log file name, for example: access.log-20240310.gz. This feature makes it easy to identify the date of the log file.
• mail [email protected]: Sends the rotated log file to the specified email address.
• olddir /path/to/archive: Moves rotated log files to a different archive directory, instead of keeping them in the same directory as the original file.

Special attention should be paid to the prerotate and postrotate scripts. For applications that require a reload or restart to recognize new log files (e.g., Nginx, Apache, PHP-FPM…), sending a signal or running a restart command is crucial. If omitted, logs might continue to be written to the old, renamed file, or worse, stop logging altogether!

Testing & Monitoring Logrotate

Dry Run

This is an indispensable step before deploying any new configuration to production. Always perform a dry run to ensure everything works as expected without affecting the system:

sudo logrotate -d /etc/logrotate.conf

This command will simulate logrotate‘s operation without actually changing any files. You will see detailed steps of the checks performed, rotation decisions, and commands that would be executed.

To test a separate configuration file, for example for Nginx:

sudo logrotate -d /etc/logrotate.d/nginx

Force Run

In cases where immediate testing or urgent disk space release is needed:

sudo logrotate -f /etc/logrotate.conf

This command forces logrotate to run and perform log rotation according to the configuration, regardless of whether the frequency conditions (daily, weekly) have been met. Exercise caution when using -f in a production environment, especially if the system has many active logs. Always thoroughly check the postrotate scripts to avoid service interruptions.

Check Status and Schedule

Logrotate usually runs periodically via cron or systemd timer. You can check its activity history at:

• /var/lib/logrotate/status: This file stores information about the last time each log file was rotated.

sudo cat /var/lib/logrotate/status

You can also view system logs to see how logrotate ran and concluded:

journalctl -u logrotate.timer
journalctl -u logrotate.service

For older systems or traditional cron jobs:

sudo grep logrotate /var/log/syslog

Or check the /var/log/cron file on CentOS/RHEL.

Monitoring and Troubleshooting

If logrotate is not working as expected, follow these steps:

1. Check status file: View /var/lib/logrotate/status to see the last time logs were rotated.
2. Check permissions: Ensure logrotate has read/write permissions to the log directory and execute permissions for postrotate/prerotate scripts.
3. View logrotate logs: Use journalctl or grep on system logs to find errors.
4. Run a dry run: This is always the most effective way to simulate and detect issues before they affect the actual system.

Log management is an essential part of system operation. By understanding and configuring logrotate correctly, you not only maintain server stability but also avoid sleepless nights troubleshooting issues caused by full disk drives.