Posted inVMware

GPU Sharing over the Network with VMware vSphere Bitfusion: Optimizing Performance and Maximizing Resource Utilization

VMware tutorial - IT technology blog
VMware tutorial - IT technology blog

The Reality: When Expensive GPUs Go to Waste

AI and Deep Learning practitioners are all too familiar with this scenario: a team requests a high-end server equipped with several NVIDIA A100 or RTX 3090 cards worth tens of thousands of dollars. However, reality often falls short of expectations. When you assign a GPU directly to a Virtual Machine (VM) via DirectPath I/O, that VM takes exclusive ownership of the resource.

The result? While a Data Scientist is busy coding or preprocessing data, the GPU sits idle. Meanwhile, a colleague wanting to “borrow” a bit of processing power to quickly test a model is out of luck because the GPU is locked down. I’ve seen research labs where GPUs run at 100% capacity for only 2 hours a day. For the remaining 22 hours, that expensive hardware does nothing but heat up the server room. This is a massive waste of resources.

Why Is GPU Virtualization So Difficult?

The root of the problem lies in the hardware architecture. GPUs were not originally designed for flexible sharing like CPUs or RAM. Historically, we had two main options, both with critical drawbacks:

  • PCI Passthrough: Provides 100% performance, but only one VM can use one GPU. To move it to another machine, you must shut down the VM and reconfigure everything manually. It’s a very tedious process.
  • NVIDIA vGPU (GRID): This allows partitioning a GPU into profiles. However, NVIDIA Enterprise licensing is extremely expensive (often over $500/user/year) and only supports specific Tesla/Quadro card series.

Bitfusion emerged to solve this dilemma: How can a VM on Server A borrow GPU power from Server B over the network?

Bitfusion: A Software Solution for a Hardware Problem

Instead of intervening at the physical layer, Bitfusion operates on a Client-Server model. It acts as a proxy layer for CUDA function calls. The machine with the physical GPU acts as the Server, while the machine running the AI code is the Client. The two connect via Ethernet or RDMA.

The real selling point of Bitfusion is its transparency. You don’t need to change a single line of AI code. TensorFlow or PyTorch still perceive the GPU as if it were locally attached. Most importantly, you can partition the GPU by RAM capacity (for example, allocating 20% of a 40GB A100 to a small task).

Step 1: Deploying the Bitfusion Server (Appliance)

First, download the Bitfusion OVF file from the VMware homepage. The Bitfusion Server is essentially a VM running a highly optimized version of Photon OS.

  1. Deploy the OVF to vCenter. Ensure you select the Host that has the physical GPU cards installed.
  2. In the VM hardware configuration, add a PCI Device and select the corresponding NVIDIA card.
  3. Important Note: Enable the Reserve all guest memory feature. Without this step, Passthrough will not function.
  4. After booting up, access the Bitfusion web interface via port 443 to activate it and connect to vCenter.

Step 2: Configuring the Client on a Linux VM

This is the Data Scientist’s workspace. I usually prefer Ubuntu 20.04 or 22.04. A major advantage is that you don’t need to install heavy NVIDIA Drivers here; you only need the much lighter Bitfusion Client package.

Add the repo and install using the following commands:

# Install the client package (assuming you already have the deb file)
sudo dpkg -i bitfusion-client-ubuntu2004_4.0.0-11_amd64.deb
sudo apt-get install -f

Next, connect this client machine to the server using a security token from vCenter:

# Check connection to the centralized GPU cluster
bitfusion list_servers

If the GPU list appears with an “Active” status, your system is ready for action.

Step 3: Running AI Applications Over the Network

This is where the magic happens. Instead of running python train.py, you simply wrap it with a Bitfusion command. Suppose you want to borrow 1 GPU and use only 50% of its memory:

# Borrow 1 GPU, limit to 50% memory to leave room for others
bitfusion run -n 1 -p 0.5 python3 my_model.py

The Bitfusion Client intercepts CUDA API calls and sends them over the network to the Server for processing. With a 10Gbps network infrastructure, latency is typically under 10%, making it almost indistinguishable from a locally installed card.

Field Experience: Don’t Let the Network Become a Bottleneck

During real-world deployment, I’ve identified three critical considerations:

  • Network Bandwidth: Absolutely do not use a 1Gbps network. It will make even the most powerful GPU run at a snail’s pace. Use at least 10Gbps. If possible, deploy Mellanox with RDMA (RoCE) for peak performance.
  • Version Synchronization: The Server and Client should run the same version. I once lost an entire afternoon because the Server was on version 4.0 while the Client was on 3.5, causing them to be unable to see each other.
  • Jumbo Frames: Enable MTU 9000 across the entire switch infrastructure. This reduces CPU overhead when transmitting large data blocks between the Client and Server.

After switching to Bitfusion, our team’s hardware utilization skyrocketed. Instead of each person hogging a card, 4-5 people can now share a single high-end GPU for smaller tasks. If you’re struggling with the high cost of GPU procurement, Bitfusion is currently the most practical solution.

Share:
Tags: