LiteLLM Proxy: One API for Every Model

Every team that ships more than one AI feature eventually hits the same wall: five services, five different SDKs, five sets of API keys scattered across .env files, and no single place to see how much you're spending. A litellm proxy fixes this by putting a single OpenAI-compatible endpoint in front of every model you use, whether that's a hosted API like OpenAI or Anthropic, or a self-hosted model running on vLLM or Ollama. Your application code stops caring which provider answered the request. This guide walks through installing the proxy, wiring up multiple providers, adding virtual keys and budgets, setting up fallbacks and caching, and deploying it with Docker.

What the LiteLLM Proxy Actually Does

LiteLLM ships as two things: a Python SDK you can import directly into an app, and a proxy server (also called the LiteLLM gateway) that runs as its own process. The proxy is the part most teams care about in production. It exposes an /chat/completions, /embeddings, and /completions API that matches the OpenAI request and response schema exactly. Point any OpenAI-compatible client at it, swap the base URL, and every downstream call now flows through the proxy instead of hitting a provider directly.

Behind that single endpoint, the proxy holds a routing table (defined in a config.yaml file) that maps model names to actual provider credentials. When a request comes in asking for gpt-4o, the proxy looks up which provider and API key that name maps to, translates the request into that provider's native format if needed, and returns a normalized response. Ask for claude-sonnet in the next request and the same client, same code path, same error handling, just works, because the proxy handles translation internally.

This matters for three reasons that come up in almost every production AI stack:

• Provider outages stop being your outage. If a primary model starts timing out, the proxy can fall back to a secondary model automatically, without your application code knowing anything happened.
• Cost and usage tracking gets centralized. Instead of grepping through five provider dashboards, spend per team, per API key, and per model shows up in one place.
• Key rotation and access control move out of application code. Virtual keys issued by the proxy can be revoked or rate-limited without touching a single deployed service.

Installing and Running the LiteLLM Proxy

The proxy is a Python package. Install it with the proxy extra so you get the FastAPI server and its dependencies:

pip install 'litellm[proxy]'

Once installed, you can start the proxy against a single model without any config file, useful for a quick smoke test:

litellm --model gpt-4o-mini

By default this starts a server on http://0.0.0.0:4000. Confirm it's alive:

curl http://localhost:4000/health/liveliness

That single-model mode is fine for a demo, but real usage means defining a config file that lists every model you want routed through the gateway. Create config.yaml:

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Start the proxy pointing at that file:

export OPENAI_API_KEY=sk-...
export LITELLM_MASTER_KEY=sk-my-master-key
litellm --config config.yaml --port 4000

The master_key is the admin credential for the proxy itself. Keep it separate from any provider key, it's what authenticates requests to the proxy's own API and dashboard, not to OpenAI or Anthropic.

Configuring Multiple Providers in One Proxy

The real value shows up once model_list has entries from more than one provider. Here's a config that routes three different backends behind three different model names:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: gemini-pro
    litellm_params:
      model: gemini/gemini-2.5-pro
      api_key: os.environ/GEMINI_API_KEY

  - model_name: local-llama
    litellm_params:
      model: ollama/llama3
      api_base: http://localhost:11434

Notice the model_name field is arbitrary, it's the name your application will ask for, while litellm_params.model is the provider-specific identifier LiteLLM translates internally. This indirection is what lets you rename models, swap providers, or run A/B tests without changing a single line of client code. If you decide next month that claude-sonnet should actually route to a newer model version, you edit the config and restart the proxy. No application redeploy needed.

You can also register the same model_name multiple times with different underlying providers, which sets up automatic load balancing:

model_list:
  - model_name: production-model
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY_1

  - model_name: production-model
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY_2

Requests for production-model now round-robin across both keys, which is a simple way to double your effective rate limit against a single provider.

Routing Requests Through the Proxy

Once the proxy is running, any OpenAI SDK client can talk to it by changing the base_url. In Python:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-my-master-key",
)

response = client.chat.completions.create(
    model="claude-sonnet",
    messages=[{"role": "user", "content": "Explain vector databases in two sentences."}],
)

print(response.choices[0].message.content)

The same pattern works in JavaScript with the official OpenAI SDK:

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:4000",
  apiKey: "sk-my-master-key",
});

const response = await client.chat.completions.create({
  model: "gemini-pro",
  messages: [{ role: "user", content: "List three uses of embeddings." }],
});

console.log(response.choices[0].message.content);

Or skip the SDK entirely and hit it with plain curl, which is handy for debugging:

curl http://localhost:4000/chat/completions \
  -H "Authorization: Bearer sk-my-master-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Say hello in five languages"}]
  }'

Every one of these calls returns a response shaped exactly like an OpenAI chat completion, regardless of which provider actually generated it. Streaming, function calling, and JSON mode all work through the same interface, as long as the underlying model supports them.

Load Balancing and Fallbacks

Production traffic needs resilience, and this is where the litellm proxy earns its keep over hand-rolled provider clients. Add a router_settings block to configure retry and fallback behavior:

router_settings:
  routing_strategy: usage-based-routing
  num_retries: 3
  timeout: 30
  fallbacks:
    - gpt-4o: ["claude-sonnet", "gemini-pro"]

With this config, a request for gpt-4o that times out or returns a 429 automatically retries against claude-sonnet, then gemini-pro, before giving up. Your application receives one clean response or one clean error, never the intermediate retries. routing_strategy controls how the proxy picks among duplicate model_name entries: usage-based-routing balances by current token throughput, least-busy picks the deployment with the fewest active requests, and simple-shuffle just round-robins.

For latency-sensitive paths, set a lower per-request timeout and a smaller retry count so a failing provider doesn't stack up delay:

router_settings:
  timeout: 8
  num_retries: 1

Adding Authentication with Virtual Keys

The master key is too powerful to hand to every service and script. Instead, issue scoped virtual keys through the proxy's own API. First make sure the proxy is backed by a database (Postgres works well) so keys persist across restarts:

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

With the database connected, create a virtual key restricted to specific models and a spend cap:

curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-my-master-key" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["gpt-4o-mini", "claude-sonnet"],
    "max_budget": 50,
    "duration": "30d",
    "metadata": {"team": "growth"}
  }'

The response includes a new sk-... key scoped to only those two models, capped at a fifty-dollar budget over thirty days. Hand that key to the growth team's service. If it leaks or the team is shut down, revoke it without touching anything else:

curl http://localhost:4000/key/delete \
  -H "Authorization: Bearer sk-my-master-key" \
  -H "Content-Type: application/json" \
  -d '{"keys": ["sk-the-leaked-key"]}'

This is the pattern that makes the proxy worth adopting even for a single-provider setup: key management becomes an API call instead of a redeploy.

Tracking Spend and Setting Budgets

Every request through the proxy gets logged with token counts and computed cost, based on LiteLLM's built-in pricing table for known models. Query spend per key:

curl http://localhost:4000/key/info?key=sk-the-growth-team-key \
  -H "Authorization: Bearer sk-my-master-key"

Or pull aggregate spend across a time window for reporting:

curl "http://localhost:4000/spend/logs?start_date=2026-06-01&end_date=2026-06-30" \
  -H "Authorization: Bearer sk-my-master-key"

You can also set budgets at the team level, not just per key, which is useful when several keys share one cost center:

curl http://localhost:4000/team/new \
  -H "Authorization: Bearer sk-my-master-key" \
  -H "Content-Type: application/json" \
  -d '{
    "team_alias": "growth-eng",
    "max_budget": 500,
    "budget_duration": "30d"
  }'

Once a team or key hits its budget, further requests return a budget-exceeded error instead of silently continuing to bill. This alone has saved teams from the classic runaway-loop bill shock, where a bug in a retry loop burns through thousands of dollars overnight before anyone notices.

Logging and Observability

The proxy supports pluggable logging callbacks so every request, response, latency, and cost figure can flow into whatever observability stack you already run. Enable a callback in the config:

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

Set the matching environment variables for whichever backend you pick, and every call through the proxy now shows up there automatically, no instrumentation needed inside your application. This works for tools like Langfuse, Datadog, and generic webhook callbacks, so you can route litellm proxy logs into the same dashboards your team already watches for the rest of the stack.

For quick local debugging without an external tool, just turn on verbose console logging:

litellm --config config.yaml --detailed_debug

This prints the exact provider request and response for every call, which is the fastest way to figure out why a particular model is returning an unexpected error.

Caching Responses

Repeated identical prompts, common in chatbots answering FAQ-style questions or in test suites hitting the same fixture prompt, don't need to hit a provider twice. Enable caching in the config:

litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT

With Redis configured, the proxy hashes the request (model, messages, and relevant parameters) and returns a cached response for an exact match, skipping the provider call entirely. This cuts both latency and cost for workloads with repetitive prompts. If Redis isn't available, an in-memory cache works for single-instance deployments, just drop the redis params and use type: local instead.

Deploying the LiteLLM Proxy with Docker

For anything beyond local development, run the proxy as a container next to your database and cache. A minimal docker-compose.yml:

version: "3.9"
services:
  litellm-proxy:
    image: ghcr.io/berriai/litellm:main-latest
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - DATABASE_URL=${DATABASE_URL}
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    depends_on:
      - postgres
      - redis

  postgres:
    image: postgres:16
    environment:
      - POSTGRES_PASSWORD=litellm
    volumes:
      - pgdata:/var/lib/postgresql/data

  redis:
    image: redis:7

volumes:
  pgdata:

Bring it up:

docker compose up -d

Behind a load balancer, run multiple replicas of the litellm-proxy service pointed at the same Postgres and Redis instances, so virtual keys, spend data, and cache all stay consistent across replicas. Health-check /health/liveliness for readiness probes, and expose /metrics if you want Prometheus scraping the proxy directly.

Common Pitfalls

A few issues come up repeatedly when teams first stand up a litellm proxy:

• Forgetting to scope virtual keys. Handing out keys with access to every model in model_list defeats the purpose of having budgets and access control in the first place. Always pass an explicit models list when generating a key.
• Mismatched model names between config and client code. If the application asks for gpt4o but the config defines gpt-4o, the proxy returns a clean "model not found" error. Keep model_name values consistent, and consider a shared constants file between the config and any client wrapper.
• No database in production. Running without database_url means virtual keys and spend tracking reset on every proxy restart. Fine for local testing, a real liability in production.
• Skipping timeouts on fallback chains. A fallback list with no timeout set can let a single slow provider hold up the entire retry chain. Set timeout explicitly in router_settings.
• Treating the master key like a regular key. The master key can create and delete other keys, change budgets, and read every log. Store it in a secrets manager, not in application .env files that get shipped to multiple services.

FAQ

What is the difference between the LiteLLM SDK and the LiteLLM proxy? The SDK is a Python library you import directly, useful for scripts and notebooks where you control the process. The proxy is a standalone server that exposes an OpenAI-compatible HTTP API, meant to sit in front of multiple applications and services regardless of what language they're written in.

Does the litellm proxy support streaming responses? Yes. Streaming works the same way it does with a direct OpenAI client, set stream: true in the request and read server-sent events from the response. This applies across providers, so a streaming request routed to Claude or Gemini through the proxy behaves identically to one routed to OpenAI.

Can I run the proxy without a database? Yes, for local development or single-node testing. Without database_url, the proxy still routes requests and applies rate limits defined in config.yaml, but virtual keys and spend history won't persist across restarts.

How does cost tracking work for self-hosted models like Ollama? Self-hosted models typically have no per-token price, so the proxy logs token counts and latency but reports zero or a custom cost if you define one manually in the litellm_params for that model entry.

Is the proxy only useful for teams with many providers? No. Even a single-provider setup benefits from centralized budgets, virtual keys that can be revoked independently of application deploys, and a consistent logging layer. Adding a second provider later becomes a config change instead of a code change.

What happens if every fallback model in a chain fails? The proxy returns the final provider's error back to the client after exhausting the fallback list and retry count, so the application still gets a single clear failure to handle rather than a hung request.