Accelerating Llama-3 with TensorRT-LLM on Ubuntu: From Installation to Production – ITFROMZERO

Table of Contents

Solving Real-World Latency Challenges

My team’s Llama-3 chatbot system used to get overloaded during peak hours. Even when running vLLM on an A100 cluster, the throughput (TPS) plummeted as the number of users increased. Latency hit the 10-second mark for each response, making the user experience terrible. After hours of investigation, I realized the bottleneck was not fully utilizing the specialized instruction sets on NVIDIA GPUs.

To put Large Language Models (LLMs) into production, a simple plug-and-play setup isn’t enough. You need to dive deep into the GPU compilation layer. That’s where TensorRT-LLM comes in. If you want to boost response speeds from 5-10 tokens/sec to over 50 tokens/sec, this is currently the ultimate solution.

Why is TensorRT-LLM Significantly Faster?

TensorRT-LLM isn’t just a model loading library like Transformers. It acts as a compiler, compiling the entire neural network architecture into an engine optimized specifically for each GPU architecture. Instead of just theoretical explanations, let’s look at three key techniques that drive its high performance:

In-flight Batching: This technique allows new requests to be processed as soon as an old one finishes, rather than waiting for the entire batch to complete.
Quantization (FP8/INT8): Reducing model size to run faster on H100 or A100 cards. This saves up to 40% VRAM while maintaining near-identical accuracy.
Kernel Fusion: Merging hundreds of small, separate matrix operations into a single block. This significantly reduces memory access time (VRAM throughput).

Setting Up a Standard Environment

A hard-learned lesson: never install directly on the host machine because TensorRT-LLM is extremely sensitive to library versions. A single mismatch in a CUDA patch can break the system. Docker is the safest choice. Here is the configuration I’ve tested for stability in production:

Operating System: Ubuntu 22.04 LTS
Hardware: NVIDIA Ampere or newer (RTX 3090, 4090, A10, A100…)
Driver: Version 535 or newer

First, install the NVIDIA Container Toolkit so Docker can communicate with the GPU:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

3-Step Deployment Workflow

1. Prepare the Checkpoint

You cannot directly load .safetensors files from HuggingFace into the engine. We need to convert them to the TensorRT-LLM intermediate format. Use the official NVIDIA (NGC) Docker image to ensure compatibility:

# Fetch source code and enter the container
git clone https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM

# Convert Llama-3-8B weights
python3 examples/llama/convert_checkpoint.py \
    --model_dir ./llama-3-8b-hf \
    --output_dir ./llama-3-8b-ckpt \
    --dtype float16

2. Build the Optimized Engine

This is the most critical stage. The trtllm-build command analyzes the current hardware to generate the most optimized executable. Pay attention to the max_batch_size parameter; if set too high, VRAM will be fully occupied even without active requests.

trtllm-build --checkpoint_dir ./llama-3-8b-ckpt \
    --output_dir ./llama-3-8b-engine \
    --gemm_plugin float16 \
    --max_batch_size 4 \
    --max_input_len 2048

3. Run with Triton Inference Server

To serve stable APIs to thousands of users, I always use Triton Server instead of running manual Python scripts. Triton manages queues and GPU resource allocation very efficiently. You need to organize your directory according to NVIDIA’s model_repository structure for the server to recognize the engine.

docker run --gpus all --rm -p 8000:8000 \
    -v $(pwd)/model_repository:/models \
    nvcr.io/nvidia/tritonserver:24.03-trtllm-py3 \
    tritonserver --model-repository=/models

Notes on Avoiding Out of Memory (OOM) Errors

In practice, OOM errors often occur when the KV Cache consumes too much memory. If your card has only 24GB VRAM like the RTX 3090, consider using Weight-Only Quantization. This technique compresses the Llama-3-8B model from 15GB down to about 8GB, giving you more headroom to increase max_input_len for long conversations.

Additionally, always verify performance using the genai-perf tool. In our team’s tests, enabling gemm_plugin increased matrix calculation speed by 25% compared to default settings. Don’t overlook small optimization flags; combined, they make a massive difference in throughput.

Conclusion

Deploying TensorRT-LLM requires meticulous attention to environment setup and parameter configuration. However, the reward is a system capable of handling high loads with lightning-fast responses. If you’re building AI products for end-users, deep-level optimization like this is a mandatory step to save infrastructure costs and improve service quality.