Configuring NVMe over TCP on vSphere 8: Maximizing Storage Performance at Minimal Cost

The Nightmare of Storage Bottlenecks

I once managed an 8-host ESXi cluster running iSCSI over a 10GbE network. Everything worked fine until the ERP system and SQL databases started getting overloaded. Latency frequently spiked to 15-20ms, causing applications to respond sluggishly.

At the time, upgrading to Fibre Channel (FC) was the obvious solution. However, the cost of HBA cards and dedicated optical switches ran into tens of thousands of dollars. Fortunately, starting from vSphere 7 Update 3 through vSphere 8, VMware has supported NVMe over TCP (NVMe/TCP). This technology solves the performance puzzle using existing Ethernet infrastructure.

NVMe over TCP vs. iSCSI and Fibre Channel

Why should you care about NVMe/TCP? Let’s look at the technical differences:

• iSCSI: Uses the SCSI command set from the HDD era. It has only a single command queue, causing bottlenecks during multi-threaded processing and consuming significant CPU resources for packet encapsulation.
• Fibre Channel: High speed and stable, but expensive and requires specialized administrative skills.
• NVMe/TCP: Uses the modern NVMe command set with support for up to 64,000 queues. This reduces latency by up to 50% compared to iSCSI and fully leverages the bandwidth of today’s 25GbE/100GbE switches.

Real-world Performance Comparison

Why I Chose NVMe over TCP for Production Systems

Cost savings is the most obvious advantage. Instead of buying new SAN switches, I repurposed an existing pair of 25GbE switches. The real-world performance after the migration was impressive. Data backup tasks saw a 30% reduction in time, and users no longer complained about application freezes during peak hours.

Deployment Steps on vSphere 8

Before configuring, ensure that the switch has Jumbo Frames (MTU 9000) enabled. This is crucial for achieving optimal storage bandwidth.

Step 1: Set up VMkernel Networking

Never share the Management network card for storage. Separate the traffic to ensure security and performance.

1. Access the vSphere Client, select Host -> Configure -> Networking.
2. Create a new VMkernel Adapter on a dedicated Distributed Switch or Standard Switch.
3. Assign a static IP address in the same range as the Storage Controller. Note: vSphere 8 automatically detects NVMe traffic, so you don’t need to select any special services at this step.

Step 2: Enable Software NVMe over TCP Adapter

In vSphere 8, this adapter does not appear automatically and must be added manually.

1. Go to the Storage Adapters section in the Configure tab.
2. Click Add Storage Adapter and select Add Software NVMe over TCP adapter.
3. Select the correct Physical NIC that you prepared for storage.

The system will create a virtual adapter (e.g., vmhba64). This is the host’s primary communication port with the storage array.

Step 3: Define the Storage Controller

Now, we will establish the connection to the storage array.

1. Select the newly created NVMe/TCP adapter and switch to the Controllers tab.
2. Click Add Controller and enter the Storage Target IP.
3. The default port is usually 4420. Click Discover Controllers to let the system scan automatically.

Step 4: Verify Connectivity via CLI

To ensure everything is running smoothly, SSH into the ESXi host and run the following command:

esxcli nvme controller list

If the State column shows Connected, you have succeeded. If it shows Disconnected, double-check the routing or firewall on the switch.

Step 5: Create Datastore

The NVMe drives will now appear as standard storage devices. Simply go to New Datastore, select the VMFS 6 format, and proceed with formatting. NVMe formatting is extremely fast, almost instantaneous.

Real-world Tips for System Stability

After some time in operation, I’ve gathered three important takeaways. First, use network cards that support RDMA if possible to further reduce CPU load. Second, always adhere to VMware’s High Performance Plugin (HPP) policy. This is a specific optimization mechanism for high-speed devices.

Finally, don’t forget to update the network card (NIC) firmware. In an NVMe/TCP environment, an outdated driver can cause a datastore disconnect (All Paths Down) in just seconds.

NVMe over TCP is no longer the technology of the future; it is the solution for today. If you’re looking to upgrade your system without the budget for Fibre Channel, this is the shortest path to achieving maximum performance.