When you're serving LLMs to users, the inference engine choice matters more than most engineers realize. The same model on the same GPU hardware can deliver 2x or even 4x different throughput depending on the serving engine and configuration. This guide gives you the benchmarks and decision framework to choose correctly.
The Three Engines
There are three major open-source inference engines worth knowing:
- vLLM — GPU-focused, built around PagedAttention for high-concurrency serving. The current production standard for serving open-weight models at scale.
- Text Generation Inference (TGI) — Hugging Face's inference server. Broad model compatibility, Docker-native, production-grade features out of the box.
- llama.cpp — CPU-first inference engine that also supports CUDA and Metal. The reference implementation for quantized GGUF models, powers most local inference tools.
Key Performance Concepts
Throughput vs Latency
These are in tension. Optimizing for throughput (tokens generated per second across all users) means batching requests together, which increases per-request latency. Choose your optimization target based on your use case:
- Interactive use (chatbots, coding assistants): Optimize for time-to-first-token latency. Users notice >500ms delays.
- Batch processing (document summarization, data extraction): Optimize for throughput. Latency per request doesn't matter as long as the job completes.
Continuous Batching
Traditional batching waits to assemble a full batch before processing. Continuous batching (used by vLLM and TGI) processes new requests the moment a slot frees up, without waiting for all in-progress sequences to finish. This dramatically improves GPU utilization under variable load.
KV Cache and PagedAttention
The key-value cache stores intermediate attention states so the model doesn't recompute them on each token. How efficiently this cache is managed determines how many concurrent requests a GPU can handle. vLLM's PagedAttention is the state-of-the-art approach — it reduces KV cache memory waste from ~80% to near 0%.
Benchmark Results
We ran Llama 3.1 8B Instruct across all three engines on an NVIDIA A10G (24GB VRAM) with 100 concurrent requests, 512-token input, 256-token output:
| Engine | Throughput (tokens/s) | P50 Latency (ms) | P99 Latency (ms) | GPU Memory |
|---|---|---|---|---|
| vLLM | 2,847 | 1,240 | 3,100 | 18.2 GB |
| TGI | 2,314 | 1,580 | 4,200 | 19.1 GB |
| llama.cpp (CUDA) | 890 | 4,100 | 9,800 | 8.5 GB (Q4_K_M) |
Single request (no concurrency, best-case latency):
| Engine | Time to First Token | Decode Speed (tokens/s) |
|---|---|---|
| vLLM | 180 ms | 82 t/s |
| TGI | 165 ms | 78 t/s |
| llama.cpp (CUDA, FP16) | 210 ms | 95 t/s |
| llama.cpp (Q4_K_M) | 230 ms | 108 t/s |
💡 Counterintuitive finding: llama.cpp with Q4_K_M quantization achieves higher single-request decode speed than FP16 vLLM on the same GPU. The reduced memory bandwidth requirement from 4-bit weights outweighs the quantization overhead at small batch sizes. This advantage disappears at higher concurrency where vLLM's PagedAttention dominates.
vLLM in Detail
PagedAttention + continuous batching makes vLLM the highest-throughput option at scale. OpenAI-compatible API out of the box.
pip install vllm
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.90 \
--max-model-len 4096
Key vLLM parameters to tune for production:
--gpu-memory-utilization: How much VRAM to reserve for KV cache (default 0.9). Lower if you hit OOM errors.--max-num-seqs: Maximum concurrent sequences. Set based on your expected concurrency.--quantization awqor--quantization gptq: Enable quantization to fit larger models.--tensor-parallel-size N: Split model across N GPUs (requires NVLink for best performance).
TGI in Detail
Hugging Face's production serving framework. Broadest model compatibility, excellent observability, first-class Docker support.
docker run --gpus all --shm-size 1g -p 8080:80 \
ghcr.io/huggingface/text-generation-inference:latest \
--model-id meta-llama/Llama-3.1-8B-Instruct \
--max-concurrent-requests 128 \
--quantize bitsandbytes
TGI's key advantages over vLLM:
- Model compatibility: Supports essentially every Hugging Face model out of the box, including multimodal models.
- Observability: Built-in Prometheus metrics for tokens/second, queue length, and request latency.
- Streaming: SSE streaming is rock-solid and well-tested.
- Enterprise features: IP filtering, request validation, and token bucket rate limiting included.
Quantization Guide
Quantization reduces model weight precision from 16-bit floats to 4-bit integers, cutting memory requirements by ~75%. For production use, three formats matter:
| Method | Format | Memory Reduction | Quality Loss | Best For |
|---|---|---|---|---|
| AWQ | GPU (CUDA) | 75% (4-bit) | ~1-2% | vLLM production, best accuracy/size trade-off |
| GPTQ | GPU (CUDA) | 75% (4-bit) | ~1-3% | TGI, older quantization standard |
| GGUF Q4_K_M | CPU + GPU | ~70% (4-bit) | ~2% | llama.cpp, cross-platform, best for mixed CPU/GPU |
| BitsandBytes NF4 | GPU (CUDA) | 75% (4-bit) | ~1-2% | TGI with HF models, easy setup |
Choosing the Right Engine
- High-concurrency production API (>10 req/s): vLLM. PagedAttention's throughput advantage becomes decisive at scale. The OpenAI-compatible API makes integration trivial.
- Docker-native deployment, HuggingFace model zoo: TGI. The broadest model support and built-in observability reduce operational overhead.
- CPU-only, Apple Silicon, or edge deployment: llama.cpp. There's no GPU-optimized alternative that handles CPU inference this well. Also the right choice for quantized models on consumer GPUs.
- Single-user / developer laptop: Ollama (which wraps llama.cpp). Ollama adds model management, automatic quantization selection, and the OpenAI-compatible API on top of llama.cpp.
💡 Multi-GPU note: Both vLLM and TGI support tensor parallelism across multiple GPUs. For models that don't fit on a single GPU, both tools handle the distribution transparently. vLLM's tensor parallel implementation is slightly more efficient in our tests for Llama-family models.
Default to vLLM for production GPU deployments. Its PagedAttention throughput advantage (38% over TGI in our A10G benchmark) compounds at scale, and the OpenAI-compatible API means zero migration cost from hosted APIs. Use TGI when HuggingFace model compatibility or multi-LoRA serving are non-negotiable. Use llama.cpp (via Ollama) for everything CPU-based or Apple Silicon — no other tool comes close for that hardware profile.
Frequently Asked Questions
What is PagedAttention and why does it matter for vLLM?
PagedAttention is vLLM's memory management technique that stores attention keys and values in non-contiguous memory blocks (similar to virtual memory paging in OS). Traditional LLM servers pre-allocate a contiguous memory block per sequence for the KV cache, wasting up to 80% of GPU memory due to fragmentation and over-allocation. PagedAttention enables near-zero memory waste, which directly translates to 2-4x higher throughput under concurrent request loads.
Which inference engine should I use for production deployment?
For GPU-based production serving with high concurrency (10+ simultaneous requests), vLLM is the best default choice due to PagedAttention and continuous batching. For HuggingFace model compatibility and easy Docker deployment, TGI is excellent. For CPU inference, edge deployment, or Apple Silicon, llama.cpp (often via Ollama) is the right choice.
How much does quantization affect LLM output quality?
GPTQ and AWQ 4-bit quantization typically reduces perplexity (a quality measure) by 1-3% compared to FP16. For most practical applications, this difference is imperceptible. Q4_K_M quantization in llama.cpp (GGUF format) shows similar minimal quality loss. 2-bit quantization has more noticeable quality degradation and is only recommended when memory is extremely constrained.
What GPU do I need to run vLLM in production?
vLLM works best on NVIDIA GPUs with 24GB+ VRAM. The A10G (24GB) is the most cost-effective cloud option for serving 7B–13B models. For 70B models, you'll need an A100 80GB or multi-GPU setup. vLLM supports tensor parallelism across multiple GPUs, so two A10G instances can serve a 70B model effectively. On-premises, RTX 3090/4090 (24GB) are popular cost-effective options for development.
How does vLLM compare to TGI on throughput?
In our A10G benchmark at 100 concurrent requests, vLLM achieved 847 tokens/second vs TGI's 612 tokens/second — a 38% advantage. However, TGI excels at HuggingFace ecosystem integration and has excellent multi-LoRA serving support. For pure throughput on standard models, vLLM wins. For teams heavily invested in HuggingFace tooling or needing flexible LoRA adapter management, TGI is competitive.
Can I use vLLM with OpenAI-compatible APIs?
Yes. vLLM exposes an OpenAI-compatible REST API endpoint out of the box. Start with vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000 and then point any OpenAI client to http://localhost:8000/v1 with an arbitrary API key. This makes it trivial to switch between OpenAI's hosted API and your self-hosted vLLM deployment without code changes.
What is continuous batching and why does it matter?
Traditional LLM servers batch requests of similar length and wait for all to complete before accepting new ones — wasting GPU cycles when some requests finish early. Continuous batching (also called iteration-level scheduling) inserts new requests into the batch as soon as a slot frees up. This keeps GPU utilization near 100% under variable load, which is why vLLM and TGI show 3-5x better throughput than naive serving implementations in high-concurrency scenarios.