Self-Hosting Open LLMs: A Practical Guide

Self hosting LLMs means running an open weight model (Llama, Qwen, Mistral, DeepSeek, or similar) on hardware you control instead of calling a provider's API. Teams do it for three reasons: data never leaves their network, cost per token drops once volume is high enough, and they get to pick exactly which model version runs in production without a vendor deprecating it under them. This guide walks through the real decision points: which serving stack to use, how to size hardware, how to quantize without wrecking quality, and where teams usually get burned.

When self hosting LLMs actually makes sense

Before touching any infrastructure, run the numbers. Self hosting wins when you have sustained, predictable throughput, when the workload is a narrow task a smaller fine-tuned model handles as well as a frontier model, or when data residency rules block you from sending prompts to a third party API at all.

It loses when your traffic is spiky, when you need frontier-level reasoning that only the largest hosted models deliver, or when your team has nobody who wants to own GPU capacity planning, driver updates, and 2am OOM crashes. A rough rule that holds up across most teams: if you are spending under a few thousand dollars a month on API calls, hosted APIs are cheaper once you account for engineer time. Above that, and if usage is steady rather than bursty, self hosting starts to pay for itself, especially at 24/7 utilization where idle GPU time is the enemy.

Also weigh the hidden cost: someone has to patch CVEs in your inference server, rotate quantized weights when a better checkpoint ships, and answer the page when a node falls over during a traffic spike. Self hosting LLMs is not "set up once and forget it." It is closer to running any other stateful production service, except the failure modes are new and the tooling is younger.

Picking a model

Open weight models fall into a few useful buckets for production work:

• General instruction-following: Llama-family and Qwen-family models cover most chat and agent use cases. Larger sizes reason better; smaller sizes are cheaper to serve and faster to respond.
• Coding-focused: several open releases are specifically tuned on code and score close to proprietary models on coding benchmarks, at a fraction of the serving cost.
• Long-context and retrieval-heavy tasks: check the model card for the actual trained context length, not the architectural maximum. A lot of open models claim 128k+ context but degrade badly past the length they were actually trained and evaluated on.
• Mixture-of-experts (MoE) models: these activate only a subset of parameters per token, so a model with a huge total parameter count can serve at the latency of a much smaller dense model, at the cost of needing more total VRAM to hold all the experts.

Do not pick a model off a leaderboard alone. Pull down the top two or three candidates and run your own eval set, ideally the exact prompts your product sends today. Benchmark scores do not tell you how a model handles your system prompt, your tool-calling schema, or your domain-specific edge cases.

Serving stack: vLLM, TGI, and llama.cpp

Three serving engines cover almost every production deployment.

vLLM is the default choice for GPU serving at any real scale. It implements PagedAttention for efficient KV cache memory management, continuous batching so requests do not block each other, and an OpenAI-compatible API out of the box. Start it like this:

pip install vllm

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

That single command gives you a drop-in replacement for the OpenAI chat completions endpoint. Point your existing SDK at http://your-host:8000/v1 and swap the model name.

Text Generation Inference (TGI), from Hugging Face, is the other mature option. It has strong support for quantized models and multi-GPU tensor parallelism, and integrates cleanly if your team already lives in the Hugging Face ecosystem for model management.

docker run --gpus all -p 8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --max-input-length 4096 \
  --max-total-tokens 8192

llama.cpp is the right tool when you are running on CPU, on Apple Silicon, or on a single consumer GPU with limited VRAM. It is not built for high-concurrency production traffic the way vLLM is, but for a single-tenant internal tool or a local dev/staging environment, it is the simplest path:

./llama-server -m model-q4_k_m.gguf --host 0.0.0.0 --port 8080 -c 8192 -ngl 999

The -ngl 999 flag offloads as many layers as fit onto the GPU; drop it or lower the number if you are running CPU-only or VRAM-constrained.

For most teams standing up a real service: start with vLLM. It has the widest adoption, the best throughput per dollar of GPU, and the largest surface of production war stories already written up by other teams, which matters when something breaks at 2am.

Sizing hardware and quantization

VRAM is the constraint that decides everything else. A rough formula for a dense model at full precision (fp16/bf16):

VRAM needed (GB) ≈ params (billions) x 2

An 8B model needs roughly 16GB just for weights, before accounting for the KV cache, which grows with context length and concurrent requests. Add 20-40% headroom for the KV cache and activation memory in a production setting with real concurrency.

Quantization is how most teams make this affordable. The common formats:

• FP16/BF16: full precision, best quality, most VRAM. Use this only if VRAM is not the constraint.
• INT8: roughly half the VRAM of fp16, quality loss is usually negligible for most tasks.
• INT4 (GPTQ, AWQ, or GGUF Q4_K_M): roughly a quarter of the VRAM of fp16. Quality loss becomes noticeable on tasks that need precise reasoning or long chains of tool calls, less noticeable on straightforward chat or extraction tasks.

A practical sizing example: a 70B parameter model at fp16 needs around 140GB of VRAM, which means multiple high-end GPUs. The same model quantized to INT4 fits in roughly 35-40GB, which is a single high-memory GPU. That difference is usually what decides whether a team can self host a large model at all, or has to drop down to a smaller one.

Always re-run your eval set after quantizing. Do not assume a quantized model behaves the same as the original; measure it on the tasks you actually care about, not a generic benchmark.

Deployment: containers, orchestration, and autoscaling

Package the serving engine into a container and pin exact versions of the driver, CUDA toolkit, and the serving library. GPU inference stacks are notoriously version-sensitive, a driver bump that a base image quietly pulls in can silently change latency or throughput.

For anything beyond a single box, run on Kubernetes with a GPU-aware scheduler, or use a managed GPU orchestration platform that handles bin-packing GPUs across nodes for you. Key production concerns:

• Cold start: loading a large model checkpoint from disk into VRAM can take minutes. If you autoscale by spinning up new pods on demand, users hit that latency directly. Keep a warm pool of minimum replicas instead of scaling to zero for anything latency-sensitive.
• Health checks: a model server that is technically up but has not finished loading weights will fail requests. Use a readiness probe that checks the actual inference endpoint, not just that the process is listening on a port.
• Batching under load: continuous batching engines like vLLM handle concurrent requests well, but throughput and per-request latency trade off against each other as batch size grows. Load test at your expected concurrency before you trust a latency number from a single-request benchmark.
• Multi-GPU tensor parallelism: for models too large for one GPU, both vLLM and TGI support splitting the model across GPUs with a --tensor-parallel-size flag. This adds inter-GPU communication overhead, so throughput does not scale linearly with GPU count.

Monitoring what matters

Standard infra metrics (CPU, memory, request count) are not enough for an LLM server. Track these specifically:

• Time to first token (TTFT): how long a user waits before anything streams back. This is what users actually perceive as "speed."
• Tokens per second, per request and in aggregate: aggregate throughput tells you if you are GPU-bound; per-request throughput tells you if a specific request is being starved by batching.
• GPU memory utilization and KV cache occupancy: if the KV cache fills up, new requests queue or get rejected. This is the most common cause of production incidents in self-hosted LLM serving.
• Queue depth and request rejection rate: tells you when you are under-provisioned before users start complaining.
• Output quality drift: if you swap checkpoints, quantization formats, or serving engines, run your eval set again. A silent quality regression is worse than a visible outage because nobody notices until a customer does.

Wire these into whatever you already use for observability (Prometheus plus Grafana is the common pairing; vLLM exposes Prometheus metrics natively). Do not build a bespoke dashboard from scratch when the serving engine already ships the instrumentation.

Where teams get burned

A few failure patterns show up again and again in self-hosted LLM deployments:

• Underestimating KV cache growth. A model that serves fine at low concurrency falls over under real traffic because the KV cache scales with both context length and number of concurrent requests, not just model size.
• Skipping the eval step after quantization. Teams quantize to save cost, ship it, and only discover a quality regression weeks later from a support ticket, not from their own testing.
• No fallback path. If your only inference path is a single self-hosted cluster, a driver crash or an OOM event takes down your whole product. Keep a fallback to a hosted API for critical paths, even if you rarely use it.
• Ignoring license terms. "Open weight" does not always mean unrestricted commercial use. Some model licenses cap usage by monthly active users or restrict certain use cases. Read the actual license file, not just the model card summary, before you build a business on top of it.
• Treating this as a one-time setup. Model checkpoints improve, serving engines ship performance fixes, and CVEs land in inference stacks. Self hosting LLMs is ongoing operational work, budget for it the same way you would budget for owning a database in production.

FAQ

Is self hosting LLMs actually cheaper than using an API? Only at sustained, high, predictable volume, and only if you count engineer time honestly. At low or spiky volume, hosted APIs almost always win because you are not paying for idle GPU capacity or the person maintaining the deployment.

Which open model should I start with? Start with whichever model your eval set scores best on, tested at the quantization level you actually plan to run in production. Do not pick based on a general leaderboard; pick based on your own prompts and tasks.

Do I need multiple GPUs to self host an LLM? Not for smaller models. An 8B-class model fits comfortably on a single modern GPU, even quantized down further for headroom. Multi-GPU setups become necessary once you move into the 70B-plus parameter range at higher precision.

How much does quantization hurt quality? INT8 is usually close to lossless for most tasks. INT4 introduces measurable degradation on tasks requiring precise multi-step reasoning or tool calling, but is often fine for chat, summarization, and extraction. Always test on your own eval set rather than trusting a general claim.

Can I self host and still fall back to a hosted API? Yes, and you should for anything customer-facing. Route through an abstraction layer (many teams use an OpenAI-compatible interface on both sides) so a self-hosted cluster outage can fail over to a hosted provider without a code change.

What is the minimum team size to run this in production responsibly? There is no hard number, but someone needs to own GPU capacity planning, driver and CVE patching, and incident response for the inference stack specifically. If nobody on the team wants that job, that is a strong signal to stay on a hosted API until you do.