Deploying Hugging Face TGI on Docker: Optimizing Throughput and Latency for Production LLMs

A 2:00 AM Postmortem: When an LLM Server Collapsed Under Request Traffic

At exactly 2:00 AM, PagerDuty went off relentlessly. The entire AI backend cluster was flashing crimson. On the Datadog dashboard, p99 latency spiked straight from 420ms to 45 seconds. Hundreds of incoming requests immediately threw HTTP 504 Gateway Timeouts.

All 24GB of VRAM on our NVIDIA A10G GPU hit a 99% ceiling. At the time, the service was still running on a DIY FastAPI wrapper around the vanilla Transformers library. When traffic surged from 15 to 120 req/s, the server queued and processed requests sequentially. Severe KV cache fragmentation brought down the entire cluster, freezing it completely.

The hard-earned lesson from that night: never throw a development setup into production to shoulder real traffic. To serve hundreds of concurrent connections on an LLM, a dedicated inference engine is non-negotiable.

Comparing 3 Production LLM Deployment Approaches

That night, we weighed three options to salvage our GPU cluster:

• Traditional FastAPI + Transformers: Wrapping Hugging Face’s pipeline in a custom API. It takes only 15 minutes to set up and is easy to modify, making it great for prompt testing or internal demos.
• vLLM via Docker Container: The popular open-source engine powered by PagedAttention. It manages KV cache seamlessly, delivers exceptional throughput under heavy batching, and enjoys massive community support.
• Hugging Face Text Generation Inference (TGI) on Docker: Hugging Face’s production-grade solution built with a hybrid Rust and C++/CUDA architecture. It comes out of the box with Continuous Batching, PagedAttention, FlashAttention-2, Tensor Parallelism, an OpenAI-compatible endpoint, and high-performance gRPC.

Pros and Cons Analysis

1. FastAPI + Transformers

Its biggest strength is rapid development and simple debugging. However, its fatal flaw is the lack of continuous batching. When 10 users send requests simultaneously, the second user must wait until the first user generates their final token. GPU utilization erratically swings between 100% and 0%. VRAM bloats uncontrollably, leading to chronic CUDA Out-Of-Memory (OOM) errors, especially when compared against dedicated frameworks like LangServe for serving production endpoints.

2. vLLM

vLLM eliminates memory waste through its PagedAttention paging mechanism. Its token generation speeds are among the best available today. The main drawback involves enterprise integration: if you need to pull weights from private Hugging Face Hub repositories, enforce strict token authentication, or track token-level streaming metrics, vLLM requires additional manual runtime configuration.

3. Hugging Face TGI

TGI features a highly pragmatic hybrid architecture. Its Rust-based router and scheduler eliminate dispatch bottlenecks with zero-overhead scheduling, while deeply optimized C++/CUDA kernels handle inference. The official Docker image supports FlashAttention-2 and quantization formats like AWQ, GPTQ, and BitsAndBytes without requiring checkpoint conversions. On the downside, the Docker image is quite heavy (around 15-20GB) and requires GPUs with Turing or Ampere architectures and newer.

Why We Chose TGI to Rescue Our Production Cluster

In the middle of an incident where the server needed to be revived immediately, TGI checked all three critical boxes:

1. Pre-packaged Official Docker Image: No wasting time debugging driver mismatches, PyTorch conflicts, or wheel compilation issues. As long as the host has the NVIDIA Container Toolkit installed, you can spin it up with a single command.
2. Battle-Tested Continuous Batching: TGI avoids static request batching. The moment Request A outputs its final token, that freed slot immediately admits Request B mid-flight. This yielded a 4x to 5x throughput improvement over our legacy FastAPI script.
3. Proactive VRAM Protection: TGI strictly caps input/output token counts and queue concurrency. It automatically cancels timed-out requests before the GPU can run out of memory. Teams scaling beyond single instances often pair this setup with solutions like Ray Serve for high-performance inference clusters.

Hands-On Guide: Deploying TGI with Docker

Step 1: Verify the Host Environment and NVIDIA Container Toolkit

First, verify that your host has the proper drivers installed and that Docker can access the GPU. Run the following command to check:

docker run --rm --gpus all nvidia/cuda:12.1.1-base-ubuntu22.04 nvidia-smi

Once you see the complete GPU status table (A10G, A100, or RTX 4090), you are ready to pull the TGI image.

Step 2: Create a Cache Directory and Launch the Container

Avoid redownloading tens of gigabytes of model weights every time the container restarts by mounting a cache directory from the host. Here is the deployment command for Qwen/Qwen2.5-7B-Instruct that rescued our team that night:

mkdir -p /data/hf_cache

docker run -d \
  --name tgi-server \
  --gpus all \
  --shm-size 1g \
  -p 8080:80 \
  -v /data/hf_cache:/data \
  -e MODEL_ID=Qwen/Qwen2.5-7B-Instruct \
  -e MAX_CONCURRENT_REQUESTS=128 \
  -e MAX_INPUT_TOKENS=2048 \
  -e MAX_TOTAL_TOKENS=4096 \
  -e MAX_BATCH_PREFILL_TOKENS=4096 \
  ghcr.io/huggingface/text-generation-inference:2.4.0

Step 3: Tune Parameters to Prevent Server Crashes

Depending on your available VRAM, adjust the following environment variables:

• --shm-size 1g: Strictly required. Rust and Python workers use this shared memory for high-speed IPC. Without it, the container will crash due to exhausted POSIX shared memory.
• MAX_INPUT_TOKENS & MAX_TOTAL_TOKENS: Defend against payloads with multi-thousand-token prompts that overflow the KV cache.
• MAX_BATCH_PREFILL_TOKENS: Caps concurrent token compute during the prompt prefill phase. On an 80GB A100 GPU, you can increase this to 8192 to accelerate responsiveness.
• MAX_CONCURRENT_REQUESTS: Queue ceiling. Requests exceeding this threshold immediately receive an HTTP 429 status code to protect the GPU.

Step 4: Monitor Startup Logs

The container takes several dozen seconds to load the model into VRAM and warm up kernels. Monitor the process using:

docker logs -f tgi-server

When the logs print Connected along with Router started at 0.0.0.0:80, the server is ready to accept traffic.

Step 5: Test the Endpoint with cURL

TGI exposes an OpenAI-compatible /v1/chat/completions endpoint. You can measure Time-To-First-Token (TTFT) with a straightforward cURL command, or benchmark performance against high-throughput alternatives like Groq Cloud:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "tgi",
    "messages": [
      {"role": "system", "content": "You are a DevOps expert assisting with incident troubleshooting."},
      {"role": "user", "content": "How do I check for a GPU memory leak?"}
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }'

Step 6: Stream Tokens with a Python Client

To render real-time typewriter-style text in your user interface, stream outputs from the /generate_stream endpoint:

import requests
import json

url = "http://localhost:8080/generate_stream"
payload = {
    "inputs": "Briefly explain the Continuous Batching mechanism:",
    "parameters": {
        "max_new_tokens": 200,
        "temperature": 0.2
    }
}

response = requests.post(url, json=payload, stream=True)

for line in response.iter_lines():
    if line:
        decoded_line = line.decode('utf-8')
        if decoded_line.startswith("data:"):
            data = json.loads(decoded_line[5:])
            print(data.get("token", {}).get("text", ""), end="", flush=True)
print()

Production Troubleshooting Playbook

• OOM immediately upon startup: If a 7B model exceeds 15GB of VRAM in FP16 precision, pass -e QUANTIZE=awq or -e QUANTIZE=bitsandbytes-nf4. This drops required VRAM down to roughly 5.5GB–6GB. For local testing without dedicated GPUs, developers often prefer running SLMs locally with Ollama to avoid memory overhead.
• Sudden spikes in p99 latency: This is typically caused by clients sending oversized prompts. Restrict request body sizes at your upstream reverse proxy (Nginx or Traefik) to drop them early. Setting up proactive tracing with AgentOps monitoring solutions can also help catch upstream bottleneck anomalies before they trigger outages.
• Serving 70B models across multi-GPU setups: Set -e NUM_SHARD=4 (for 4 GPUs). TGI handles tensor parallelism over NCCL automatically without requiring manual torchrun configurations.

Forty minutes after completing the migration to TGI, post-incident metrics taken at 3:00 AM showed p99 latency plummeting from 45 seconds down to 1.15 seconds. VRAM consumption stabilized at 81% under a continuous 120 req/s load. By 9:00 AM the following morning, the system breezed through peak traffic without a single OOM error.