Get Started in 5 Minutes
If you’re running LLMs with vLLM without Speculative Decoding enabled, you’re leaving 40–60% of your GPU’s potential speed on the table. This section gets it up and running immediately.
Prerequisites
You need vLLM version 0.4.0 or higher:
pip install vllm --upgrade
python -c "import vllm; print(vllm.__version__)"Starting the Server with Speculative Decoding
Run the following command to enable it right away — target model Llama 3.1 8B, draft model Llama 3.2 1B:
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--speculative-model meta-llama/Llama-3.2-1B-Instruct \
--num-speculative-tokens 5 \
--use-v2-block-manager \
--gpu-memory-utilization 0.9That’s only 2 extra lines compared to a standard vLLM command. I’ve used this setup in production and the results are quite stable — throughput increases around 1.8–2x with the Llama 3.1 8B + 1B draft combo on an A10G.
Quick Test to Verify It’s Working
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "Write a Python script to read a CSV file:",
"max_tokens": 200,
"temperature": 0
}'Note the temperature: 0 when testing — speculative decoding works best with greedy decoding.
How Speculative Decoding Works
Understanding the mechanism helps you tune it correctly rather than guess. The core bottleneck of LLM inference: every generated token requires a full forward pass through the entire model. On a GPU with an 8B model, that step consumes enormous memory bandwidth — even though the actual compute required is relatively modest.
The Draft-then-Verify Mechanism
Speculative Decoding splits inference into 2 steps:
- Draft phase: A small draft model (e.g., 1B) quickly generates N tokens in sequence — these are called “speculative tokens”.
- Verify phase: The larger target model (8B/70B) verifies all N tokens in a single forward pass.
If the draft model predicts all 5 tokens correctly, you’ve just generated 5 tokens at the cost of 1 target model forward pass + 1 draft model pass — instead of 5 separate passes. That’s why throughput increases so significantly.
What if the draft guesses wrong at token k? The target model rejects from that position, keeps the previously correct tokens, and regenerates. No incorrect tokens make it through — the output is identical to running the target model alone. This is what sets it apart from quantization or pruning, which trade off output quality.
When Is It Most Effective?
- Low temperature (0.0–0.6): draft model predictions are more likely to be correct
- Repetitive output patterns: code, markdown, technical lists
- GPU bottlenecked by memory bandwidth, not FLOPs
For creative writing at high temperatures (above 0.8), gains diminish because the draft model struggles to keep up with the randomness of the target model.
Advanced Configuration
Tuning the Number of Speculative Tokens
The --num-speculative-tokens parameter has a significant impact on real-world results:
# Conservative: 3 tokens — high acceptance rate, moderate throughput gain
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--speculative-model meta-llama/Llama-3.2-1B-Instruct \
--num-speculative-tokens 3
# Aggressive: 7 tokens — higher throughput if workload is suitable
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--speculative-model meta-llama/Llama-3.2-1B-Instruct \
--num-speculative-tokens 7Simple rule: if the draft token acceptance rate is above 70%, increase num-speculative-tokens. Below 50%, reduce it or switch to a different draft model.
Ngram-Based Speculative Decoding — No Draft Model Required
No suitable draft model or running out of VRAM? vLLM supports ngram prompt lookup — using the input context itself to predict the next tokens:
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-70B-Instruct \
--speculative-model "[ngram]" \
--num-speculative-tokens 5 \
--ngram-prompt-lookup-max 4 \
--use-v2-block-managerThis option is particularly useful for code generation and RAG — tasks where the output tends to repeat terms from the input. It requires no additional GPU memory and only needs 2 extra flags.
EAGLE Speculative Decoding
Rather than using a smaller model from the same family as the draft, EAGLE trains a dedicated draft model specifically aligned to the target model — resulting in a 15–20% higher real-world acceptance rate:
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--speculative-model yuhuili/EAGLE-LLaMA3-Instruct-8B \
--speculative-draft-tensor-parallel-size 1 \
--num-speculative-tokens 4Find EAGLE models on HuggingFace using the format EAGLE-{ModelName}. Beyond Llama 3.x, there are corresponding models for Qwen2.5 and Mistral — just search by target model name.
Multi-GPU setup
# Target model runs on 2 GPUs, draft model runs on 1 GPU
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 2 \
--speculative-model meta-llama/Llama-3.1-8B-Instruct \
--speculative-draft-tensor-parallel-size 1 \
--num-speculative-tokens 4 \
--use-v2-block-managerNote: --speculative-draft-tensor-parallel-size must be a divisor of --tensor-parallel-size.
Practical Tips
Choose a Draft Model From the Same Family
The draft model needs to share the same tokenizer and architecture as the target model for a high acceptance rate:
- Llama 3.1 8B → draft: Llama 3.2 1B or 3B
- Llama 3.1 70B → draft: Llama 3.1 8B
- Qwen2.5 7B → draft: Qwen2.5 0.5B or 1.5B
- Mistral 7B → no smaller model from the same family — use ngram instead
Draft models from a different family (e.g., Qwen as draft for Llama) typically yield acceptance rates below 30%, fail to improve speed, and can actually be slower.
Calculate GPU Memory Before Enabling
# Estimated memory requirements (fp16)
# Target model 8B: ~16 GB
# Draft model 1B: ~2 GB
# KV cache: ~4 GB
# Total: ~22 GB → requires a 24 GB GPU (RTX 3090/4090 or A10G)
nvidia-smi --query-gpu=memory.total,memory.free --format=csvRunning low on VRAM? Use int8 quantization for the draft model to save ~50% memory:
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--speculative-model meta-llama/Llama-3.2-1B-Instruct \
--speculative-model-quantization int8 \
--num-speculative-tokens 5Monitor the Real-World Acceptance Rate
vLLM exposes metrics so you can tell whether speculative decoding is actually working:
import requests
metrics = requests.get("http://localhost:8000/metrics").text
for line in metrics.split("\n"):
if "spec" in line.lower() and not line.startswith("#"):
print(line)Key metrics to watch:
vllm:spec_decode_draft_acceptance_rate: target above 0.6vllm:spec_decode_efficiency: efficiency relative to baselinevllm:generation_tokens_per_second: actual throughput
Benchmark Before Going to Production
python benchmarks/benchmark_throughput.py \
--model meta-llama/Llama-3.1-8B-Instruct \
--dataset sharegpt \
--num-prompts 200 \
--speculative-model meta-llama/Llama-3.2-1B-Instruct \
--num-speculative-tokens 5Benchmark results vary significantly by workload: code generation typically sees 2.5–3x throughput gains, conversational chat around 1.5–2x. Don’t rely on other people’s numbers to make decisions — run benchmarks with a dataset that resembles your actual production traffic to get meaningful results.
Speculative Decoding rarely gets mentioned in basic vLLM guides, but it’s one of the few techniques that lets you increase throughput without touching output quality. The only requirement: choose the right draft model and tune num-speculative-tokens based on your application’s actual input patterns.
