Real-world Problem: When a CentOS Server Disk Suddenly Runs Out of Space
IT professionals are surely familiar with the scenario: everything’s calm, then suddenly you get an alert that the server is out of disk space, or worse, the system starts to lag, and some services even crash. At that point, the first thing I do is run the df -h command to see which partition is ‘eating up’ the most space.
I remember back at my old company, we had a few CentOS 7 servers running internal services. Occasionally, a developer colleague would inadvertently push gigabytes of log files or large datasets to their home directory without realizing it. Only when the disk was full, the system started slowing down, or services crashed, would we scramble to investigate. This not only wasted time on troubleshooting but also directly impacted the work of many others.
In many companies, some servers still run CentOS 7. Migrating to AlmaLinux has been a handled challenge. However, whether it’s CentOS 7 or AlmaLinux, resource management, especially storage space, always remains a top priority. Without strict control mechanisms, an individual or group can easily monopolize too many resources. This creates imbalance and negatively impacts the entire system.
Root Cause Analysis: Why Do We ‘Lose Control’ Over Disk Space?
The main reason for the above situation is often the lack of proactive storage space management and limitation mechanisms. By default, when a user is created on CentOS (or any Linux operating system), they can store data in their home directory or any directory where they have write permissions. However, no disk space limits are applied by default.
- Lack of Proactivity: Admins often only react when a problem has already occurred (full disk) instead of implementing preventive measures beforehand.
- Unaware Users: Many users, especially those not specialized in systems, are unaware of the amount of disk space they are consuming and its impact on the overall system.
- No Automatic Control Tools: Manually checking with
du -shordf -his only a temporary solution; it doesn’t automatically prevent abuse. - Unequal Resource Allocation: Without limits, a few users can inadvertently or intentionally monopolize most of the space. This leaves other users without enough room to work.
The consequences are not limited to a full disk. It also leads to decreased system performance, the risk of data loss, and difficulties in backup and recovery. Furthermore, this can create a security vulnerability, allowing attackers to perform a Denial-of-Service (DoS) attack by filling up the disk.
Solutions (and Why They Aren’t the Best)
When facing disk space issues, several solutions are often considered:
1. Manual Checking and Reminders
This is the most common approach when starting out. Periodically, administrators use commands like du -sh /home/* to check which users are consuming the most disk space, then send emails or chat messages reminding them to clean up.
Pros: Simple, no complex configuration required.
Cons: Time-consuming, prone to oversight, reactive, not proactive in prevention. Users might forget or not cooperate, and administrators have to repeat this task constantly. Clearly, this is not a scalable solution.
2. Writing Scripts for Automatic Checking and Notification
A step forward is to develop a shell script, run periodically via a cron job, to automatically check the disk space usage of each user or directory. If a certain threshold is exceeded, the script will send an email notification to that user and the administrator.
Pros: Partially automated, reduces manual effort.
Cons: Still a reactive measure. The script only notifies; it doesn’t prevent users from creating more files. Users can continue to abuse storage until they read the email and clean up. Sometimes, deleting other users’ files requires root privileges, which isn’t always optimal or secure.
3. Implementing Disk Quota
This is the proactive and most effective solution we want to introduce. Disk Quota is an operating system feature that allows administrators to limit the storage space (blocks) and the number of files (inodes) that a user or group can consume on a specific filesystem.
Pros:
- Proactive prevention: Prevents users from exceeding limits, avoiding full disk situations from the outset.
- Fair allocation: Ensures each user/group has a defined portion of storage space, preventing a few individuals from over-consuming.
- System stability: Helps maintain server performance and stability.
- Easy to manage: Once configured, management becomes simpler and less effort-intensive.
The Best Approach: A Guide to Implementing Disk Quota on CentOS
To implement Disk Quota on CentOS, we will follow these steps. We will use the example of limiting disk space on the /home partition, which typically contains user data.
Step 1: Check and Install the Quota Package
First, ensure that the quota package is installed on your system.
rpm -qa | grep quota
If there’s no output or the package is missing, you need to install it:
sudo yum install quota -y
Step 2: Enable Quota on the Filesystem
For Quota to work, we need to edit the /etc/fstab file to add the usrquota (for users) and grpquota (for groups) options to the partition where quota should be applied.
Open the /etc/fstab file using your preferred editor (e.g., vi or nano):
sudo vi /etc/fstab
Locate the configuration line for the partition where Quota needs to be applied (e.g., /home). This line typically looks like this:
/dev/mapper/centos-home /home xfs defaults 0 0
Or if it’s ext4:
/dev/sda1 /home ext4 defaults 1 2
You need to add usrquota and grpquota to the defaults section, separated by commas. If other options are already present, add them after those:
/dev/mapper/centos-home /home xfs defaults,usrquota,grpquota 0 0
After editing, save and exit the file. For the changes to take effect, you need to remount the filesystem. The safest way is to restart the server. However, to avoid downtime, you can remount:
sudo mount -o remount /home
Note: If you are using an XFS filesystem (which is often the default on CentOS 7 and later), the usrquota and grpquota options are sufficient. For ext4, you need to ensure kernel support.
Step 3: Create the Quota Database File
Now, we need to create the database files for Quota on the configured partition. The quotacheck command will scan the filesystem and create these database files. These files are aquota.user and aquota.group (or quota.user and quota.group for older ext4), located in the root directory of the partition.
sudo quotacheck -cug /home
-c: Create new database files.-u: Scan and create a database for users.-g: Scan and create a database for groups.
You can add -v to see detailed scanning progress.
Step 4: Configure Quota Limits for Users/Groups
This is the most crucial step: setting limits for users or groups. We will use the edquota command.
Configure for users:
sudo edquota -u <username>
For example, to configure for user itfromzero:
sudo edquota -u itfromzero
This command will open an editor (usually vi) with content similar to the following:
Disk quotas for user itfromzero (uid 1001):
Filesystem blocks soft hard inodes soft hard
/dev/mapper/centos-home 0 0 0 0 0 0
Explanation of columns:
blocks: Current number of blocks used by the user.soft: Soft limit. Users can exceed this limit for a specified grace period.hard: Hard limit. Users cannot exceed this limit. If the hard limit is reached, they will not be able to write any more data.inodes: Current number of files/directories (inodes) used by the user.inodes soft/inodes hard: Similar to blocks, but applies to the number of files/directories.
You can set limits in KB, MB, GB by adding the suffixes K, M, G. For example, to set a soft limit of 1GB and a hard limit of 1.2GB for blocks for user itfromzero. Additionally, a soft limit of 50,000 files and a hard limit of 60,000 files:
Disk quotas for user itfromzero (uid 1001):
Filesystem blocks soft hard inodes soft hard
/dev/mapper/centos-home 0 1G 1.2G 0 50000 60000
Save and exit the editor.
Configure for groups:
Similar to users, you can set limits for a group:
sudo edquota -g <groupname>
For example:
sudo edquota -g developers
The configuration method is similar to that for users.
Configure Grace Period:
The grace period is the amount of time a user is allowed to exceed their soft limit before the soft limit becomes enforced as a hard limit. You can configure it using the command:
sudo edquota -t
This will open a file similar to:
Grace period before enforcing soft limits for users:
Time units may be: days, hours, minutes, seconds
Filesystem Block grace period Inode grace period
/dev/mapper/centos-home 7days 7days
You can change 7days to 3days, 2hours, etc. For example, a common grace period is 5days to give users enough time to clean up.
Step 5: Enable Quota
Once the limits are configured, you need to enable Quota on the partition:
sudo quotaon /home
To disable Quota, use quotaoff /home.
Step 6: Check Quota Status
To check if Quota is active and limits are applied correctly, you can use the following commands:
Check overall status for all partitions:
sudo repquota -a
The output will show the Quota status for all enabled partitions.
Check Quota for a specific user:
sudo quota -s itfromzero
This command will display detailed information about the disk space and file count that user itfromzero is using compared to their limits.
Disk quotas for user itfromzero (uid 1001):
Filesystem blocks soft hard inodes soft hard
/dev/mapper/centos-home 4K 1G 1.2G 1 50K 60K
If the user exceeds the soft limit, the blocks or inodes column will show a ‘+’ sign, and the grace period will count down.
Tips and Best Practices from Real-world Experience
- Apply strategically: Quota is often most effective when applied to partitions containing user data or specific application data, e.g.,
/home,/var/www,/srv. Avoid applying it to the root partition/unless absolutely necessary, as this can complicate system management. - Educate users: Even if you’ve set up Quota, informing and explaining these limits to users is crucial. They need to understand the reasons and how to check their own disk usage to avoid work interruptions.
- Monitor regularly: Although Quota operates automatically, you should still perform periodic checks with
repquota -ato get an overview and make adjustments if necessary. - Use
edquota -pto create templates: If you need to apply the same limits to multiple users, you can configure a template user. Then, use the commandedquota -p <template_user> <user1> <user2> ...to copy the configuration. This method saves a lot of time. - XFS Quota: On CentOS 7 and later versions, the default filesystem is often XFS for
/. Therefore, if working with XFS, you might usexfs_quotainstead of the traditionalquotacommands. The configuration is fundamentally similar, but the commands will differ slightly. For example:xfs_quota -x -c 'limit bsoft=1g bhard=1.2g itfromzero' /home. - Consider Inode limits: Sometimes, a user might not consume much disk space but creates a large number of small files (e.g., cache files, session logs). Limiting inodes helps control this situation, preventing waste of filesystem resources.
Conclusion
Managing Disk Quota on CentOS is an essential skill for any system administrator, especially in multi-user or shared server environments. By proactively limiting storage space, we not only prevent full-disk incidents but also ensure fair resource allocation, maintaining performance and stability for the entire system.
Don’t wait until your server ‘cries for help’ due to lack of space before addressing the problem. Implement Disk Quota today for a healthier and more manageable system!
