Quick start: Unbottlenecking Your Server in 5 Minutes
It’s 2 AM, and PagerDuty is blowing up. A 64GB RAM PostgreSQL server suddenly stalls: query latency spikes from 20ms to 3,500ms. Even with no traffic increase, system CPU (%sy) hits 65%. Firing up perf top reveals the culprit immediately: the CPU is bogged down in page table walks and trapped in kernel spinlocks.
Is your system allocating tens of gigabytes to buffer cache or Java heap? If Linux still has to slice all that RAM into tiny 4KB blocks, the CPU will get exhausted. Here are three quick steps to enable static HugePages (2MB) and turn things around.
1. Check the OS HugePage Size
grep -i hugepagesize /proc/meminfo
# Standard x86_64 architecture typically returns:
# Hugepagesize: 2048 kB (2MB)
2. Allocate HugePages on the Fly via sysctl
Suppose your database requires 16GB of shared_buffers. With a 2MB page size, you need 8,192 pages (16,384MB / 2MB). Add an extra ~5% headroom for auxiliary processes and round up to 8,600 pages:
# Allocate 8,600 2MB pages directly in active RAM without a reboot
sudo sysctl -w vm.nr_hugepages=8600
# Verify whether the kernel has allocated enough pages
grep -i hugepages /proc/meminfo | grep -E "HugePages_Total|HugePages_Free"
3. Configure Memory Locking Limits (memlock)
Database processes require permission to pin memory into RAM to prevent it from being swapped out. If managing services via systemd (Ubuntu 20.04+, RHEL 8+), create an override file:
sudo systemctl edit postgresql
# Add the following two lines to the override file:
[Service]
LimitMEMLOCK=infinity
For daemons managed via init scripts or regular shell sessions, add the following to /etc/security/limits.conf:
postgres soft memlock unlimited
postgres hard memlock unlimited
Restart your database. Check %sy in top again, and you will see system CPU usage drop significantly.
Technical Deep Dive: Why HugePages Solves the Root Cause
The Page Table Bottleneck and TLB Misses
By default on Linux x86_64, RAM is broken down into 4KB pages. The CPU uses a Page Table to translate virtual application addresses into physical memory addresses. To keep lookups fast, the CPU relies on an extremely fast hardware cache called the TLB (Translation Lookaside Buffer).
Problems arise when application memory grows substantially:
- A PostgreSQL server with 64GB RAM for buffers requires up to 16,777,216 4KB pages. The 4-level page table itself consumes over 1.3GB of RAM just to maintain the address mapping directory.
- The TLB on modern CPUs (like Intel Xeon or AMD EPYC) holds only a few thousand entries. That means the TLB can only cover a few tens of megabytes of actual memory.
- When queries scan through large volumes of data, TLB misses cascade. On every miss, the CPU burns dozens of clock cycles walking through the 4-level page table. As a result,
%syspikes, and CPU cores stall waiting on memory.
Switching to 2MB HugePages (512 times larger) completely changes the game. That 64GB of RAM now requires only 32,768 pages. The page table shrinks to just a few megabytes. The TLB cache is now capable of covering nearly the entire hot dataset, driving TLB misses down close to zero.
On our 128GB RAM PostgreSQL cluster dedicated to nightly financial reporting, switching to static HugePages cut query execution time from 45 minutes down to 28 minutes.
Static Standard HugePages vs. Transparent HugePages (THP)
These two concepts are frequently confused in production operations:
- Standard HugePages (Static): Pre-allocates a fixed memory pool. This memory is pinned in RAM, cannot be swapped out, and is accessible only to processes that explicitly request it.
- Transparent HugePages (THP): Automatically collapses 4KB pages into 2MB chunks at runtime. While convenient in theory, the background daemon
khugepagedtriggers aggressive memory compaction during runtime defragmentation. It acquires memory locks, stalling database threads for anywhere from several hundred milliseconds to several seconds.
Battle-tested advice: Disable THP entirely and configure Standard HugePages for all database workloads, Redis, and Kafka.
Production Deployment Walkthrough
1. Permanently Disable Transparent HugePages
Check the current THP status on your server:
cat /sys/kernel/mm/transparent_hugepage/enabled
# If this returns [always] madvise never -> THP is enabled!
Completely disable THP via GRUB. Open /etc/default/grub and add the flag to the kernel parameters:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash transparent_hugepage=never"
Update the bootloader and schedule a server reboot:
# On Ubuntu / Debian:
sudo update-grub
# On RHEL / CentOS / Rocky Linux:
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
2. Calculate and Persist HugePages at the OS Level
Accurate sizing formula:
Number of pages (nr_hugepages) = ceil(Required Application RAM / Hugepagesize) + Reserve Headroom (~3-5%)
Example for PostgreSQL: A server with shared_buffers = 32GB needs (32 * 1024) / 2 = 16384 pages. Adding roughly 600 buffer pages for headroom, we settle on 17,000 pages.
Persist this setting in /etc/sysctl.d/99-hugepages.conf:
vm.nr_hugepages = 17000
# Allow the database user's group to allocate shared memory via Hugetlb
vm.hugetlb_shm_group = 999 # Replace 999 with the actual GID of the postgres user (find with: id -g postgres)
Reload sysctl configuration:
sudo sysctl --system
3. Enable HugePages at the Application Layer
PostgreSQL
Open postgresql.conf and adjust the parameter:
# Defaults to 'try'. Change to 'on' to force the service to use HugePages.
# If pages are insufficient, the DB will fail to start rather than silently degrading on 4KB pages.
huge_pages = on
Restart PostgreSQL:
sudo systemctl restart postgresql
Oracle Database
Oracle automatically maps the SGA into HugePages if Linux has them allocated. Log in via sqlplus / as sysdba to strictly enforce the parameter:
-- Ensure instance only starts when sufficient HugePages are available, avoiding fallback to 4KB
ALTER SYSTEM SET USE_LARGE_PAGES=ONLY SCOPE=SPFILE;
SHUTDOWN IMMEDIATE;
STARTUP;
Java JVM (Elasticsearch, Kafka, Large-Heap Spring Boot)
For Java applications running with heaps of 16GB or higher, append the -XX:+UseLargePages flag to your startup script:
java -Xms32g -Xmx32g -XX:+UseLargePages -XX:+AlwaysPreTouch -jar service-backend.jar
Note: The -XX:+AlwaysPreTouch flag forces the JVM to touch all memory pages during startup, avoiding latency spikes when live incoming requests hit.
Troubleshooting Common Issues
1. nr_hugepages Is Set, but the Kernel Does Not Allocate Enough Pages
You run sysctl -w vm.nr_hugepages=17000, but HugePages_Total only reaches 8000. This is caused by physical memory fragmentation. After running for weeks, RAM becomes heavily fragmented, leaving the kernel unable to locate enough contiguous 2MB blocks.
How to force the kernel to defragment memory on a running system:
# 1. Flush dirty pages to disk and clear page cache
sync
echo 3 | sudo tee /proc/sys/vm/drop_caches
# 2. Compact fragmented memory blocks into contiguous spans
echo 1 | sudo tee /proc/sys/vm/compact_memory
# 3. Re-request HugePages allocation
sudo sysctl -w vm.nr_hugepages=17000
If the steps above still cannot allocate enough pages, configure hugepages=17000 as a GRUB boot parameter and reboot. This allows the kernel to reserve contiguous memory at boot before fragmentation can occur.
2. Verify If Processes Are Actually Using HugePages
Avoid relying on free -m, as it does not break down HugePages utilization. Inspect /proc/meminfo directly:
watch -n 2 'grep -i huge /proc/meminfo'
The output reports four critical metrics:
HugePages_Total: Total number of HugePages reserved by the kernel.HugePages_Free: Number of HugePages not yet mapped by any process.HugePages_Rsvd: Number of HugePages successfully reserved by applications but not yet written to.HugePages_Surp: Surplus HugePages allocated beyond the initial quota.
Signs of a successful setup: When the database starts up, HugePages_Rsvd should increase to match your configured buffer size. If HugePages_Free remains equal to HugePages_Total and HugePages_Rsvd = 0, your application is not utilizing HugePages at all. Review your LimitMEMLOCK limits or double-check the group ID in vm.hugetlb_shm_group.
