How to Manage and Limit CPU and RAM on CentOS Stream 9 Using cgroups v2 and systemd Slices – ITFROMZERO

Table of Contents

A 2 AM Incident: When a Single Process Brought Down the Entire Server

Zabbix alerts started firing incessantly at 2 AM. The entire web cluster on our CentOS Stream 9 VPS (4 vCPUs, 16GB RAM) was completely unresponsive. SSH connections repeatedly timed out, and the load average soared to 48.

After struggling through the provider’s rescue console, I finally managed to run the top command. The culprit was immediately clear: a Python queue worker with a memory leak had eaten up all 14GB of available RAM. As a result, the Linux OOM Killer was triggered and promptly terminated MariaDB and Nginx by mistake.

Allowing processes to consume resources without constraints is a major operational risk. Starting with CentOS Stream 9, Red Hat transitioned fully to cgroups v2. When combined with systemd slices, you can effectively partition and isolate resources across different application groups.

3 Approaches to Resource Management on Linux

System administrators typically consider three main approaches:

Approach 1: Manual manipulation via cgroupfs (Manually creating directories in /sys/fs/cgroup/ and assigning PIDs by hand).
Approach 2: Using ulimit or configuring limits.conf (Enforcing limits at the user session level).
Approach 3: Using cgroups v2 with systemd slices (Direct control at the systemd service unit level).

Pros and Cons of Each Solution

1. Manual Management via cgroupfs

Pros: Operates independently without relying on an init system.
Cons: Highly complex to maintain. When processes fork child processes or crash and restart, PID changes cause manual configurations to be lost.

2. Using ulimit (/etc/security/limits.conf)

Pros: Convenient for interactive user sessions via terminal.
Cons: Ineffective for daemons managed by systemd. Additionally, ulimit -v forces applications to crash immediately rather than gracefully throttling memory.

3. cgroups v2 with systemd Units & Slices

Pros: Uses a single unified hierarchy for CPU, Memory, and I/O. Processes remain strictly bound to their group regardless of how many times they restart. Multiple services can easily share a unified resource limit.
Cons: Uses new directive syntax (e.g., MemoryMax replaces MemoryLimit from cgroups v1).

Why systemd Slices + cgroups v2 Is the Optimal Choice on CentOS Stream 9

CentOS Stream 9 runs Linux kernel 5.14+ alongside systemd v250+, with cgroups v2 enabled by default. Trying to apply legacy solutions only fragments your system configuration.

With systemd slices, you can divide your server into distinct resource partitions. For example, you can group all background workers and cron jobs into background.slice (capped at 20% CPU and 2GB RAM). The remaining 80% CPU and 14GB RAM are safely reserved for system.slice, hosting your database and web server.

Step-by-Step Implementation Guide

Step 1: Verify cgroups v2 Status

First, verify that your system is actively running cgroups v2:

# Check the cgroup filesystem
stat -fc %T /sys/fs/cgroup/
# Expected output: cgroup2fs

# List enabled controllers
cat /sys/fs/cgroup/cgroup.controllers
# Expected output: cpuset cpu io memory pids

Step 2: Create a Slice to Isolate Background Services

Create a background-worker.slice file to manage background tasks:

sudo nano /etc/systemd/system/background-worker.slice

Define the resource constraints:

[Unit]
Description=Resource Limit Slice for Background Workers
Before=slices.target

[Slice]
# Limit CPU usage to 50% of a single core
CPUQuota=50%

# Soft RAM threshold: kernel actively reclaims/pages out memory above 1GB
MemoryHigh=1G

# Hard RAM threshold: cgroup OOM Killer terminates slice processes at 1.5GB
MemoryMax=1.5G

# Limit maximum processes to prevent fork bombs
TasksMax=100

Step 3: Assign the Service to the Slice

Assuming you have a service named data-worker.service, open its configuration file:

sudo systemctl edit --full data-worker.service

Add Slice=background-worker.slice under the [Service] section:

[Unit]
Description=Data Worker Background Service
After=network.target

[Service]
Type=simple
User=workeruser
Slice=background-worker.slice
ExecStart=/usr/bin/python3 /opt/worker/app.py
Restart=always

[Install]
WantedBy=multi-user.target

Reload the systemd daemon configuration and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart data-worker.service

Step 4: Quickly Limit an Individual Service (Drop-in Override)

If you only need to restrict a single service like Nginx without creating a dedicated slice, use a drop-in override file:

sudo systemctl edit nginx.service

Add the throttling directives:

### Editing /etc/systemd/system/nginx.service.d/override.conf
[Service]
# Allow Nginx to use a maximum of 2 vCPUs (200%)
CPUQuota=200%
# Cap maximum memory usage at 2GB
MemoryMax=2G
# CPU scheduling priority weight during resource contention (default is 100)
CPUWeight=200

Apply the changes:

sudo systemctl daemon-reload
sudo systemctl restart nginx

Step 5: Monitor and Verify Resource Usage

Monitor real-time resource consumption across slices with the following command:

systemd-cgtop -m

To inspect detailed memory usage and threshold limit events for the slice:

# Check current RAM usage
cat /sys/fs/cgroup/background-worker.slice/memory.current

# Check how many times MemoryHigh was hit or OOM was triggered
cat /sys/fs/cgroup/background-worker.slice/memory.events

Proactively defining resource boundaries with cgroups v2 keeps your systems stable and resilient. Even if a background process suffers a memory leak, your core services will remain fully protected.