Squeezing Every Drop of NVMe Performance for MySQL: From innodb_io_capacity to Kernel Tuning – ITFROMZERO

Table of Contents

The Story of a Ferrari in a Narrow Alley

I once managed a logging system for a large e-commerce platform with about 50,000 requests per second. When the system hit an I/O bottleneck, my boss spent thousands of dollars upgrading to Samsung Enterprise NVMe drives. The whole team was convinced performance would skyrocket because NVMe IOPS specs are dozens of times higher than old SATA SSDs.

The reality was a cold shower. After migrating the data, the iowait metric was still glowing red, and the application remained sluggish. It turned out that default MySQL settings are like driving a Ferrari in a narrow alley with a 20km/h speed limit. If you don’t know how to “unlock” it, even the high-end hardware is just for show.

Why Default MySQL Settings Waste Your Money on NVMe Drives?

The reason is simple: InnoDB’s default parameters were written back when HDDs were king.

IOPS limit too low: innodb_io_capacity defaults to only 200. Meanwhile, a mid-range NVMe drive today easily hits 300,000 – 500,000 IOPS.
Outdated Flush mechanism: InnoDB pushes data to disk too timidly, leaving the drive idle while the CPU waits for I/O.
Kernel Queues: Default Linux settings often try to reorder I/O requests to optimize for mechanical heads, which is completely redundant for NVMe flash chips.

Step 1: Force InnoDB to Work at Full Capacity

Open your my.cnf file. This is where you unleash your hardware’s power.

Configuring innodb_io_capacity

This parameter defines the I/O processing capability of the storage system. With NVMe, feel free to increase this number. Stop using the weak default value of 200.

[mysqld]
# For a single NVMe drive (e.g., Samsung 980 Pro/PM9A1)
innodb_io_capacity = 5000
innodb_io_capacity_max = 10000

# If running NVMe RAID 0 or Enterprise-grade drives
# innodb_io_capacity = 20000
# innodb_io_capacity_max = 40000

Disable innodb_flush_neighbors

On HDDs, writing adjacent data blocks helps reduce the distance the read/write head travels. With NVMe, this just wastes CPU resources for nothing. Disable it immediately.

innodb_flush_neighbors = 0

Use innodb_flush_method = O_DIRECT

By default, data is cached twice: in the InnoDB Buffer Pool and the OS Page Cache. O_DIRECT allows MySQL to write directly to the disk, freeing up RAM and minimizing write latency.

innodb_flush_method = O_DIRECT

Step 2: Tuning the Linux Kernel to “Clear the Path” for Data

The operating system is the bridge between MySQL and the hardware. If the bridge is narrow, even a powerful car cannot go fast.

Changing the I/O Scheduler

Schedulers like cfq or deadline were born to serve spinning disks. For NVMe, you should switch to none to bypass the Kernel’s intermediate sorting layer, significantly reducing latency.

# Check (replace nvme0n1 with your actual drive name)
cat /sys/block/nvme0n1/queue/scheduler

# Switch to none to achieve maximum speed
echo none > /sys/block/nvme0n1/queue/scheduler

Increase the nr_requests Queue

NVMe is excellent at parallel processing. Allow the operating system to push more requests into the queue simultaneously.

echo 1024 > /sys/block/nvme0n1/queue/nr_requests

Configure Mount with noatime

Every time you perform a SELECT, Linux defaults to recording the file access time. For a database, this is extremely wasteful. Add noatime to /etc/fstab to eliminate these redundant write operations.

/dev/nvme0n1p1  /var/lib/mysql  ext4  defaults,noatime  0 2

A Classic Mistake: Forgetting to Configure innodb_log_file_size

I once encountered a case where I/O was highly optimized, yet write speeds still stuttered every 5 minutes. Upon investigation, the culprit was innodb_log_file_size being set to only 128MB.

When the log file is full, MySQL pauses all activity to perform a checkpoint. With NVMe, you should set the log file to at least 1GB – 2GB. This extends the interval between checkpoints, leading to more stable performance under high load.

innodb_log_file_size = 2G
innodb_log_files_in_group = 2

Optimization is a Journey, Not a Destination

Don’t just copy and paste every parameter into your production server. Follow this 4-step process:

Benchmark: Use fio to measure the actual IOPS of the drive before configuration.
Kernel Tuning: Prioritize the none scheduler and noatime.
MySQL Configuration: Focus on io_capacity and O_DIRECT.
Monitoring: Run iostat -x 1. If the %util column is below 60% but the application is still slow, you still have room to increase innodb_io_capacity.

Default configurations are the enemy of performance. Be bold and make changes to leverage every cent you’ve spent on NVMe drives. The resulting application response times will prove you right!