Posted inAI

LLM Production Deployment: Building a High-Performance Inference Cluster with Ray Serve

Artificial Intelligence tutorial - IT technology blog
Artificial Intelligence tutorial - IT technology blog

Why do LLMs run smoothly locally but crash in production?

Running a Large Language Model (LLM) on a personal machine is simple. However, trouble arises when you deploy it to a server for 5-10 concurrent users. At this point, the system slows down noticeably, even freezing the GPU entirely. The issue isn’t your Python code; it’s resource management, as LLMs are extremely VRAM-intensive and require continuous parallel computation.

The three biggest hurdles you will encounter include:

  • GPU Bottleneck: A single request consumes all resources, forcing other requests to queue for dozens of seconds.
  • Budget Waste: Maintaining four A100 GPUs continuously when there are no users at night causes server costs to skyrocket.
  • Scaling Difficulties: Distributing requests across multiple physical servers is a headache if you write your own load balancer.

To solve this problem, Ray Serve is my top choice. This framework turns AI models into microservices capable of flexible auto-scaling in a Linux environment.

System Environment Preparation

You should use an Ubuntu 22.04 server with NVIDIA Drivers and CUDA Toolkit pre-installed. Instead of using Docker right away, I will guide you through installing directly on Python so you can understand the system’s operational flow.

# Update the system and install Python
sudo apt update && sudo apt upgrade -y
sudo apt install python3-pip -y

# Install Ray Serve and vLLM
pip install "ray[serve]" vllm torch

In practice, I always combine Ray Serve with vLLM. This inference engine is 10-20 times faster than HuggingFace Transformers thanks to the PagedAttention technique.

Setting up a Ray Cluster: From Single Machine to Large Scale

Ray operates on a Head node and Worker node model. The Head node acts as the “orchestrator,” while Worker nodes handle the actual GPU computations.

Initialize the Head node on the main server with the command:

ray start --head --port=6379 --dashboard-host=0.0.0.0

After running, the terminal will display an IP and token. If you have a second server, just run ray start --address='HEAD_NODE_IP:6379'. Instantly, the GPU resources from the new machine will be pooled into your compute cluster.

Coding the Inference Service Deployment

Skipping complex boilerplate setup, Ray Serve allows you to define a service via a Class with the @serve.deployment decorator. Below is an optimized code snippet to run Qwen-2.5 or Llama-3.

import ray
from ray import serve
from vllm import LLM, SamplingParams
from fastapi import FastAPI

app = FastAPI()

@serve.deployment(num_replicas=1, ray_actor_options={"num_gpus": 1})
@serve.ingress(app)
class LLMDeployment:
    def __init__(self, model_name: str):
        # Limit VRAM to avoid Out of Memory errors
        self.llm = LLM(model=model_name, gpu_memory_utilization=0.8)
        self.sampling_params = SamplingParams(temperature=0.7, max_tokens=512)

    @app.post("/generate")
    async def generate(self, prompt: str):
        outputs = self.llm.generate([prompt], self.sampling_params)
        return {"text": outputs[0].outputs[0].text}

model_path = "Qwen/Qwen2.5-7B-Instruct"
serve.run(LLMDeployment.bind(model_path))

Note the num_replicas parameter. If you own 4 GPUs, increase this number to serve more concurrent users.

Auto-scaling Configuration: The Secret to Saving 60% in Costs

In a production environment, traffic is often unstable. I usually use a YAML configuration file to let Ray Serve automatically manage the number of replicas based on actual load.

deployments:
  - name: LLMDeployment
    autoscaling_config:
      min_replicas: 1
      max_replicas: 10
      target_ongoing_requests: 5
    ray_actor_options:
      num_gpus: 1

The target_ongoing_requests: 5 configuration is crucial. If each replica is processing more than 5 concurrent requests, Ray will automatically trigger a new GPU. When demand drops, the system automatically releases GPUs to save power and server costs.

Monitoring and Performance Testing

You can monitor system health via the Ray Dashboard at port 8265. This dashboard provides visual metrics on requests per second (RPS) and VRAM usage for each GPU.

To quickly test from the terminal, use the curl command:

curl -X POST "http://localhost:8000/generate" \
     -H "Content-Type: application/json" \
     -d '{"prompt": "What is Ray Serve?"}'

If the response returns in under 1 second and the GPU utilization chart moves, your system is ready for real-world users.

Real-world Operational Experience

After many real-world deployments, I’ve gathered 3 “hard-learned” tips to keep the system from crashing:

  1. VRAM Control: vLLM defaults to 90% VRAM occupancy. If you need to run other tasks on the same GPU, adjust gpu_memory_utilization down to about 0.7 – 0.8.
  2. Set up Health Checks: LLMs sometimes hang due to CUDA driver errors. Configure health checks so Ray can automatically restart failed processes without manual intervention.
  3. Network Bandwidth: When running a multi-machine cluster, ensure nodes are connected via a minimum 10Gbps local network. This ensures model weights of dozens of GBs are transferred quickly.

Deploying this way might seem more complex than a simple Python script initially. However, it is the most sustainable path for your AI application to survive sudden spikes in user growth.

Share:
Tags: