Deploying LLM Services on Kubernetes

Running an LLM in a notebook is easy. Running an LLM Kubernetes deployment that survives traffic spikes, GPU node churn, and a 2am pager alert is a different problem entirely. This guide walks through the full stack: choosing a serving framework, packaging the model, requesting GPUs correctly, autoscaling on the metrics that actually matter, and the health checks that keep a slow-starting model from getting killed by its own liveness probe.

Everything below assumes you already have a Kubernetes cluster with at least one GPU node pool (EKS, GKE, AKS, or a bare-metal cluster with the NVIDIA device plugin installed). If GPU scheduling is new to you, don't worry, we cover the device plugin and resource requests in detail.

Why LLM workloads break normal Kubernetes assumptions

Standard web services are stateless, start in seconds, and scale linearly with CPU. LLM inference services violate all three assumptions:

• Slow startup: loading a 7B-70B parameter model from disk into GPU memory can take 30 seconds to several minutes, depending on model size and storage throughput.
• Expensive, scarce resources: a GPU node costs far more than a CPU node, and cloud providers often cap how many you can provision on demand.
• Non-linear scaling: throughput depends on batch size, sequence length, and KV cache memory, not just request count. Two requests with short prompts and two requests with 8k-token prompts consume wildly different resources.

These differences mean you can't just copy a Deployment manifest from a typical microservice and swap the image. You need GPU-aware scheduling, longer probe timeouts, and autoscaling signals tied to queue depth or GPU utilization rather than CPU.

Choosing a serving framework

Before writing any YAML, pick how the model itself gets served. The framework choice determines your container image, resource requests, and health check endpoints.

vLLM is the most common default for open-weight models in 2026. It handles continuous batching, PagedAttention for efficient KV cache memory, and exposes an OpenAI-compatible HTTP API out of the box, which simplifies client code enormously.

TGI (Text Generation Inference) from Hugging Face is a solid alternative, particularly if you're already pulling models from the Hugging Face Hub and want tight integration with their tokenizer and quantization tooling.

Triton Inference Server with the TensorRT-LLM backend is the choice when you need maximum throughput and have the engineering time to build optimized engines per model and GPU architecture. It's more work to set up but pays off at high scale.

For most teams starting out, vLLM strikes the best balance of performance and operational simplicity. The rest of this guide uses vLLM in examples, but the Kubernetes patterns (GPU requests, probes, autoscaling) apply regardless of framework.

Packaging the model into a container

Two approaches exist for getting model weights into your container: bake them into the image, or mount them from external storage at startup.

Baking weights into the image gives you immutable, versioned artifacts, which pairs well with GitOps workflows. The tradeoff is image size (a 70B model in fp16 is well over 100GB) and slower CI builds.

FROM vllm/vllm-openai:latest

# Bake model weights into the image for a fully immutable artifact
COPY ./models/llama-3-8b-instruct /models/llama-3-8b-instruct

ENV MODEL_PATH=/models/llama-3-8b-instruct

ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server", \
  "--model", "/models/llama-3-8b-instruct", \
  "--served-model-name", "llama-3-8b-instruct", \
  "--gpu-memory-utilization", "0.90", \
  "--max-model-len", "8192"]

The alternative, mounting weights from an object store or a shared filesystem, keeps images small and lets you swap model versions without a rebuild. Use an init container to pull weights before the main container starts.

initContainers:
  - name: fetch-model
    image: amazon/aws-cli:latest
    command:
      - sh
      - -c
      - aws s3 sync s3://model-artifacts/llama-3-8b-instruct /models/llama-3-8b-instruct
    volumeMounts:
      - name: model-cache
        mountPath: /models

For clusters with many GPU nodes pulling the same model repeatedly, put a caching layer (a shared NFS volume, or a tool like Fluid or Alluxio) in front of the object store so you're not re-downloading gigabytes of weights on every pod restart.

Requesting GPUs correctly

Kubernetes treats GPUs as an extended resource, not a first-class scheduling primitive like CPU or memory. You request them under resources.limits, and the NVIDIA device plugin (or the equivalent for AMD/Google TPU) advertises them to the scheduler.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
  labels:
    app: llm-inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      containers:
        - name: vllm
          image: your-registry/llama-3-8b-vllm:v1
          ports:
            - containerPort: 8000
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
            limits:
              cpu: "4"
              memory: "16Gi"
              nvidia.com/gpu: "1"
          env:
            - name: NCCL_DEBUG
              value: "WARN"
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-l4
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"

A few things worth calling out:

• GPU limits, not requests: Kubernetes only supports GPUs as a limits field, since they can't be fractionally shared the way CPU can (without MIG or time-slicing, covered below).
• Node selectors and taints: GPU nodes are almost always tainted so that CPU-only workloads don't accidentally land on expensive hardware. Your pod spec needs a matching toleration.
• One GPU per pod is the safe default. Multi-GPU inference (tensor parallelism across 2, 4, or 8 GPUs) requires the serving framework to coordinate across processes, which adds real complexity, only reach for it once a single GPU can't hold the model.

If you're running smaller models that don't need a full GPU, NVIDIA's MIG (Multi-Instance GPU) or time-slicing lets you partition a single physical GPU into multiple schedulable units. This is worth setting up once you have several small models competing for GPU capacity, since it can cut idle GPU spend significantly.

Health checks that respect slow startup

The single most common mistake in LLM Kubernetes deployments is copying default liveness and readiness probe timings from a web service template. A model that takes 90 seconds to load will get killed repeatedly by a liveness probe with a 30-second initialDelaySeconds, and the pod will crash-loop forever without ever serving a request.

Use a startupProbe to give the container room to load the model, then let liveness and readiness probes take over with tighter intervals once startup completes.

startupProbe:
  httpGet:
    path: /health
    port: 8000
  failureThreshold: 30
  periodSeconds: 10
  # allows up to 300 seconds for model load before liveness kicks in

livenessProbe:
  httpGet:
    path: /health
    port: 8000
  periodSeconds: 15
  failureThreshold: 3
  timeoutSeconds: 5

readinessProbe:
  httpGet:
    path: /health
    port: 8000
  periodSeconds: 10
  failureThreshold: 2
  timeoutSeconds: 5

If your serving framework doesn't expose a dedicated /health endpoint, add a thin sidecar or wrapper script that checks whether the model is loaded in memory (rather than just checking if the HTTP server process is up, which can respond before the model finishes loading).

Autoscaling on the right signal

CPU-based Horizontal Pod Autoscaling is close to useless for LLM inference. A pod can be pegged at 95% GPU utilization while barely touching CPU, or CPU can spike during tokenization while GPU sits idle. Scale on request queue depth or GPU utilization instead.

The cleanest path is KEDA (Kubernetes Event-Driven Autoscaling) paired with a Prometheus metric your serving framework already exports, most serving frameworks including vLLM expose queue length and GPU KV cache usage via a /metrics endpoint.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-inference-scaler
spec:
  scaleTargetRef:
    name: llm-inference
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-server.monitoring.svc.cluster.local
        metricName: vllm_num_requests_waiting
        query: avg(vllm:num_requests_waiting)
        threshold: "5"

Set cooldownPeriod generously (five minutes or more). Scaling GPU pods down and back up rapidly wastes money on model reloads and can thrash your node autoscaler, which also has to provision and deprovision GPU nodes on its own multi-minute timescale. If your workload has predictable daily patterns, consider pairing KEDA's reactive scaling with a scheduled minimum replica count during known peak hours, so you're not always racing cold starts against demand.

Routing and load balancing across replicas

A plain Kubernetes Service with round-robin routing works for a first pass, but it ignores the fact that requests have wildly different costs. A request with a 4000-token prompt ties up a GPU far longer than a 50-token one. For anything beyond a small deployment, put an inference-aware gateway in front of your pods.

Tools built for this, such as the vLLM production stack's routing layer or a custom Envoy configuration with least-outstanding-requests load balancing, route new requests to the replica with the most available KV cache capacity rather than blindly round-robining. This alone can meaningfully cut p99 latency under mixed workloads.

apiVersion: v1
kind: Service
metadata:
  name: llm-inference-svc
spec:
  selector:
    app: llm-inference
  ports:
    - port: 80
      targetPort: 8000
  type: ClusterIP

Layer an Ingress or Gateway API resource on top with request timeouts set well above your default (LLM responses, especially streamed ones, can legitimately take 30-60+ seconds), and make sure your load balancer supports HTTP streaming without buffering the entire response before forwarding it, otherwise your users lose token-by-token streaming.

Observability: what to actually watch

Standard pod metrics (CPU, memory, restart count) tell you almost nothing useful for an LLM service. Instrument and dashboard these instead:

• Time to first token (TTFT): how long a user waits before seeing any output. This is the metric users feel most directly.
• Tokens per second, per request and aggregate: your real throughput number.
• GPU memory utilization and KV cache occupancy: tells you how close you are to needing to scale out or reject requests.
• Queue depth: requests waiting for a free execution slot, your leading indicator for autoscaling and for user-facing latency degradation.
• Request duration percentiles (p50/p95/p99), not just averages, since LLM latency distributions are heavily skewed by prompt and output length.

Ship these from your serving framework's /metrics endpoint into Prometheus, then build a Grafana dashboard with TTFT and queue depth front and center. Alert on queue depth crossing a threshold sustained for several minutes, that's a much better early warning than waiting for latency SLOs to breach.

Cost control

GPU nodes are the single largest line item in most LLM deployments, so a few practical levers matter:

• Use spot/preemptible GPU nodes for stateless inference where possible, with a small on-demand baseline for reliability. Kubernetes' cluster autoscaler can mix node pools, and your Pod Disruption Budget should tolerate the occasional preemption.
• Right-size the GPU to the model. Don't run an 8B model on a GPU sized for 70B parameters; a smaller GPU class with MIG partitioning often serves several small models at a fraction of the cost.
• Quantize where accuracy tolerates it. Serving a model in 8-bit or 4-bit quantization can roughly halve or quarter memory footprint, letting you fit more concurrent requests, or a bigger model, on the same GPU.
• Scale to zero for low-traffic models using KEDA's minReplicaCount: 0, accepting the cold-start cost in exchange for not paying for an idle GPU overnight.

A minimal end-to-end checklist

Before calling an LLM Kubernetes deployment production-ready, confirm:

1. GPU resource limits and matching node selector/toleration are set correctly.
2. A startupProbe accounts for full model load time, not just process start.
3. Autoscaling triggers on queue depth or GPU utilization, not CPU.
4. Ingress/gateway timeouts and streaming support are configured for long-running, streamed responses.
5. Prometheus is scraping TTFT, tokens/sec, and queue depth, with alerts on sustained queue growth.
6. A rollout strategy (maxUnavailable: 0 with maxSurge sized to available GPU headroom) prevents a bad deploy from taking all replicas down during a model swap.

FAQ

Do I need a service mesh like Istio for LLM inference? Not required. A service mesh adds sidecar overhead and complexity that mostly benefits east-west traffic patterns between many microservices. For a small number of inference services, a plain Ingress or Gateway API resource with careful timeout configuration is usually sufficient. Reach for a mesh if you already run one for other workloads and want consistent mTLS and observability.

Can I run LLM inference on CPU-only nodes to save cost? Yes for small models (under roughly 3B parameters) or heavily quantized ones, with frameworks like llama.cpp, but expect significantly lower throughput and higher latency than GPU serving. It's a reasonable choice for low-traffic internal tools, not for user-facing production traffic at scale.

How do I handle multiple model versions running side by side? Deploy each version as a separate Deployment with a distinct served-model-name, then route traffic between them at the gateway layer using weighted routing. This lets you do canary rollouts of a new model version without touching the stable deployment, and roll back instantly by shifting the weight back.

What's the difference between horizontal pod autoscaling and cluster autoscaling here? HPA/KEDA scales the number of inference pods based on load; the cluster autoscaler (or Karpenter) scales the number of underlying GPU nodes to make room for those pods. You need both working together, since a new pod requesting a GPU is useless if no GPU node has capacity, and provisioning a new GPU node can take several minutes, which is why generous cooldowns matter.

Should I use a managed inference endpoint instead of self-hosting on Kubernetes? If your traffic is low and unpredictable, a managed endpoint avoids the operational overhead described in this guide. Self-hosting on Kubernetes pays off once you need custom models, fine-tuned weights, tight cost control at scale, or you're already running the rest of your stack on Kubernetes and want consistent tooling across services.