Serving Open LLMs with vLLM

vLLM serving is the fastest practical way to turn an open weight model like Llama, Qwen, or Mistral into a production API. If you have ever run a Hugging Face transformers model behind a Flask endpoint and watched GPU utilization sit at 20% while requests queue up one at a time, vLLM fixes exactly that problem. It batches requests continuously, manages the KV cache with a paging scheme instead of naive contiguous memory, and exposes an OpenAI-compatible HTTP server so your existing client code barely changes. This guide walks through installing vLLM, standing up a server, tuning it for real traffic, and the operational details that separate a demo from something you can put behind a load balancer.

Why vLLM Serving Beats a Naive Transformers Loop

The core problem with serving LLMs the naive way is that autoregressive generation is memory bound, not compute bound. Every request holds a growing KV cache in GPU memory for the length of its output. If you serve requests with a plain model.generate() loop, you either process one request at a time (terrible throughput) or you pad a static batch to the longest sequence (wasted memory and wasted compute on padding tokens).

vLLM solves this with two ideas:

• PagedAttention: KV cache is stored in fixed-size blocks, similar to how an operating system pages virtual memory. Blocks are allocated on demand and can be shared between sequences (useful for beam search or shared prefixes like system prompts). This means near-zero memory waste from padding or over-allocation.
• Continuous batching: instead of waiting for a whole batch to finish before starting the next one, vLLM adds new requests into the running batch as soon as a GPU slot frees up (when another sequence finishes generating). Throughput stays high even under bursty, variable-length traffic.

The practical effect: on the same GPU, vLLM commonly delivers several times the throughput of a naive Hugging Face serving loop, with lower p99 latency under concurrent load. That is the entire reason it has become the default choice for self-hosted LLM inference in 2026.

Installing vLLM and Picking a Model

Install into a fresh virtual environment. vLLM ships prebuilt wheels tied to specific CUDA and PyTorch versions, so match the install command to your GPU driver:

python3 -m venv vllm-env
source vllm-env/bin/activate
pip install --upgrade pip
pip install vllm

Check that vLLM can see your GPU and print its version:

python3 -c "import vllm; print(vllm.__version__)"
nvidia-smi

For a first run, pick a model small enough to fit comfortably on your GPU. A 7-8B parameter model in bf16 needs roughly 16GB of VRAM just for weights, plus room for the KV cache. If you are on a single 24GB card, an 8B model is a safe starting point. Larger models need either a bigger card, quantization, or tensor parallelism across multiple GPUs (covered below).

Download happens automatically from the Hugging Face Hub the first time you reference a model ID, or you can pre-fetch it:

huggingface-cli download meta-llama/Llama-3.1-8B-Instruct --local-dir ./models/llama-3.1-8b

Gated models require huggingface-cli login with a token that has accepted the model's license first.

Starting the OpenAI-Compatible Server

vLLM's server speaks the same wire protocol as the OpenAI API, which means any tool built against openai's Python or JS client, LangChain, LlamaIndex, or a raw curl script, works against it without modification. Start it with:

python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --port 8000

Key flags to understand:

• --dtype: bfloat16 is the standard choice on Ampere and newer GPUs. Use float16 on older hardware that lacks native bf16 support.
• --max-model-len: caps the context window vLLM will allocate KV cache for. Setting this lower than the model's native max (say, 8192 instead of 128000) frees a large amount of GPU memory for concurrent requests. Only raise it if you actually need long context.
• --gpu-memory-utilization: fraction of GPU memory vLLM is allowed to claim for weights and KV cache, defaults to 0.9. Lower it if you are sharing the GPU with another process.

Once the server is up, test it with a plain HTTP call:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain PagedAttention in two sentences."}],
    "max_tokens": 200
  }'

Or with the official OpenAI Python client, pointing base_url at your local server:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "List three benefits of continuous batching."}],
    max_tokens=300,
)

print(response.choices[0].message.content)

Streaming works the same way it does against the real OpenAI API, just add stream=True and iterate over the response chunks. This drop-in compatibility is what makes migrating an app from a hosted API to a self-hosted vLLM deployment mostly a config change, not a rewrite.

Tuning Throughput and Concurrency

Default settings work for a demo but leave performance on the table under real traffic. The flags that matter most for production tuning:

• --max-num-seqs: maximum number of sequences vLLM will run concurrently in a single batch. Raising this increases throughput until you hit a memory wall, then it starts hurting because the scheduler has to preempt sequences. Start around 128-256 and watch GPU memory and latency as you push it up.
• --enable-prefix-caching: turn this on when many requests share a long common prefix, like a fixed system prompt or a RAG template. vLLM will reuse the cached KV blocks for the shared prefix instead of recomputing it, which meaningfully cuts time-to-first-token for chat and RAG workloads.
• --enable-chunked-prefill: splits long prompt processing (prefill) into chunks that get interleaved with ongoing decode steps for other requests, instead of blocking the whole batch while one huge prompt is processed. This smooths out latency spikes when prompt lengths vary a lot, which is the common case in production.

A tuned launch command for a chat API with mixed prompt lengths looks like this:

python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --port 8000

Measure before and after any tuning change. vLLM ships a benchmark script that simulates concurrent traffic against a running server:

python3 -m vllm.entrypoints.benchmark_serving \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --num-prompts 200 \
  --request-rate 10

It reports throughput (tokens/sec), and latency percentiles for time-to-first-token and end-to-end completion. Run this after every config change so you are tuning against numbers, not guesses.

Scaling Across GPUs

When a model does not fit on one GPU, or when a single GPU's throughput ceiling is below what your traffic needs, vLLM supports two scaling strategies:

Tensor parallelism splits each layer's weights across multiple GPUs on the same machine. Use it when the model itself is too large for one card:

python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --dtype bfloat16 \
  --port 8000

This requires the GPUs to be connected with fast interconnect (NVLink ideally) since every layer needs an all-reduce step across the devices. Tensor parallelism across PCIe-only GPUs works but the interconnect becomes the bottleneck.

Data parallelism (running multiple independent vLLM server instances, each with a full copy of the model, behind a load balancer) is the better choice when the model fits on one GPU but you need more aggregate throughput. It scales linearly and has no cross-GPU communication overhead. In practice, for models in the 7-13B range, running N separate single-GPU vLLM instances behind nginx or a simple round-robin proxy usually beats tensor parallelism on cost-per-token.

Quantization to Fit Bigger Models

If VRAM is the constraint and you cannot add GPUs, quantization reduces the memory footprint of model weights at some cost to output quality. vLLM has native support for several quantization formats:

python3 -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Llama-3.1-8B-Instruct-AWQ \
  --quantization awq \
  --dtype float16 \
  --port 8000

AWQ and GPTQ are the most common pre-quantized formats you will find on the Hugging Face Hub, both roughly halve or quarter the memory needed for weights compared to bf16, depending on the bit width. FP8 quantization is also supported directly by vLLM on hardware that has native FP8 tensor cores, and tends to preserve quality better than 4-bit AWQ/GPTQ while still cutting memory in half versus bf16.

The tradeoff to test explicitly: quantized models are faster to load and use less memory, but you should run your own eval set against the quantized version before shipping it. Quality degradation varies a lot by model family and quantization method, it is not safe to assume a 5% quality hit across the board.

Structured Output and Tool Calling

Production apps usually need the model to return JSON matching a schema, not free text. vLLM supports structured output through its guided_json parameter, backed by grammar-constrained decoding:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

schema = {
    "type": "object",
    "properties": {
        "sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
        "confidence": {"type": "number"}
    },
    "required": ["sentiment", "confidence"]
}

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Classify: 'This product exceeded my expectations.'"}],
    extra_body={"guided_json": schema}
)

print(response.choices[0].message.content)

This is more reliable than prompting the model to "return only JSON" and hoping, because the decoder is constrained token-by-token to only produce tokens that keep the output valid against the schema. It also supports guided_choice for constraining to a fixed set of strings, and guided_regex for pattern-constrained output, both useful for classification-style tasks where you need a guaranteed output format.

Monitoring and Health Checks

Before putting vLLM behind a load balancer, wire up the endpoints it exposes for operational visibility:

• /health: returns 200 if the server is up and the model is loaded. Use this for your load balancer's health check, not a full inference call.
• /metrics: Prometheus-formatted metrics including GPU cache usage, running/waiting request counts, and token throughput. Point a Prometheus scraper at this and build a Grafana dashboard so you can see queue depth building up before latency complaints arrive.
• /v1/models: lists the loaded model, useful as a smoke test after a deploy to confirm the right model version is actually serving.

Watch the num_requests_waiting metric specifically. A sustained non-zero value means your --max-num-seqs ceiling or your GPU count is too low for the traffic you are getting, and requests are queuing behind the running batch. That is your signal to either scale out with data parallelism or raise concurrency limits, not a signal to just add a longer client-side timeout.

Common Pitfalls

A few mistakes come up repeatedly when teams move from a vLLM demo to production:

• Setting `--max-model-len` to the model's full context window by default. This reserves KV cache space for the worst case on every request slot, which crushes your effective concurrency. Set it to what your application actually needs.
• Not pinning the vLLM version. vLLM ships fast and default flag behavior occasionally shifts between releases. Pin the version in your Dockerfile and test upgrades in staging before rolling to production.
• Ignoring `--gpu-memory-utilization` when co-locating vLLM with other GPU workloads. vLLM will greedily claim GPU memory up to the configured fraction at startup. If something else needs GPU memory on the same box, you will get an out-of-memory crash, not a graceful degradation.
• Benchmarking with a single sequential client. vLLM's whole value proposition is concurrent batching. A one-request-at-a-time benchmark will make it look no faster than a naive serving loop, because you are not exercising the batching path at all. Always benchmark with realistic concurrency.

FAQ

Does vLLM support every open source model on Hugging Face? No, but coverage is broad. vLLM maintains an explicit list of supported architectures (Llama family, Qwen, Mistral, Gemma, Phi, DeepSeek, and most other popular decoder-only architectures). Check the vLLM documentation's supported models page before committing to an unusual architecture, and if a model was released recently, confirm your installed vLLM version is new enough to include it.

Can I run vLLM without a GPU? vLLM has experimental CPU support, but it is meaningfully slower and not the intended deployment target. For any workload with real traffic, a CUDA-capable GPU (or vLLM's ROCm build for AMD GPUs) is the practical requirement.

How is vLLM different from Ollama or llama.cpp? Ollama and llama.cpp are optimized for single-user, local inference, often on CPU or consumer GPUs, prioritizing ease of setup. vLLM is built for server-side, multi-tenant throughput with continuous batching and PagedAttention, and it is the better choice when you are serving many concurrent requests rather than running a model on your own laptop.

Do I need to write my own load balancer for multi-GPU data parallelism? For a simple setup, nginx or any standard HTTP load balancer round-robining across multiple vLLM server ports works fine. For larger deployments, tools like Kubernetes with a Service object, or a dedicated LLM gateway, handle health-check-aware routing so requests do not hit an instance still loading its model.

What is the difference between `--gpu-memory-utilization` and `--max-num-seqs`? --gpu-memory-utilization sets the total memory budget vLLM is allowed to use for weights and KV cache combined. --max-num-seqs caps how many sequences can run concurrently within that budget. You typically tune memory utilization once for your hardware, then tune max-num-seqs against real traffic to find the concurrency sweet spot before latency degrades.

Is quantization safe for production use? It depends on the task. For classification, extraction, or routing tasks, quantized models (AWQ, GPTQ, FP8) usually hold up well. For tasks needing precise reasoning or long-form generation quality, run a side-by-side eval against your own test set before trusting a quantized model in production, since degradation is task-dependent and not uniform.