Mocking LLM APIs in Your Tests

Mocking LLM APIs in your tests means replacing real calls to providers like Anthropic's API or OpenAI with fake responses you control, so your test suite runs fast, costs nothing, and never flakes because a model returned something unexpected. If you are building anything that calls an LLM in production code (a chatbot, an agent, a summarizer, a classifier), you need a mocking strategy before your CI bill or your test runtime gets out of hand. This article walks through the practical patterns: stubbing the HTTP layer, using provider-specific test doubles, recording and replaying real responses, and structuring your code so mocking is easy instead of painful.

Why you cannot just call the real API in tests

The naive approach is to let your tests hit the real API. It "works" for a while, then breaks down for four concrete reasons.

Cost. Every test run burns tokens. A CI pipeline that runs your suite fifty times a day, times a hundred tests that each call an LLM, adds up fast. Multiply that by every pull request and every retry, and you are paying real money to run unit tests.

Speed. A real API call takes anywhere from a few hundred milliseconds to several seconds, especially for longer completions or reasoning models. A test suite with 200 LLM-touching tests running sequentially against a live API can take fifteen minutes when it should take fifteen seconds.

Determinism. Even with temperature set to 0, model outputs can vary between runs, and providers occasionally update models behind the scenes. A test that asserts on exact output text will randomly fail for reasons that have nothing to do with your code.

Reliability and isolation. Network calls fail. Rate limits kick in. API keys expire or get rotated. Your tests should not depend on an external service being up, authenticated, and under its rate limit just to verify your prompt-building logic works.

None of this means you should never test against a real model. It means unit and most integration tests should mock the LLM call, and you reserve a small number of "live" tests (often run separately, less frequently, and gated behind an environment flag) to catch real drift.

The core pattern: isolate the LLM call behind a boundary

Before you can mock anything cleanly, your code needs a seam. If you scatter client.messages.create(...) calls throughout your business logic, every test that touches that logic needs to know about the LLM client. Instead, wrap the call in a small function or class that owns talking to the provider.

# llm_client.py
import anthropic

class LLMClient:
    def __init__(self, api_key: str, model: str = "claude-sonnet-5"):
        self.client = anthropic.Anthropic(api_key=api_key)
        self.model = model

    def complete(self, prompt: str, max_tokens: int = 1024) -> str:
        response = self.client.messages.create(
            model=self.model,
            max_tokens=max_tokens,
            messages=[{"role": "user", "content": prompt}],
        )
        return response.content[0].text

Now your business logic depends on LLMClient, not on the anthropic SDK directly.

# summarizer.py
class Summarizer:
    def __init__(self, llm_client):
        self.llm_client = llm_client

    def summarize(self, text: str) -> str:
        prompt = f"Summarize the following text in two sentences:\n\n{text}"
        return self.llm_client.complete(prompt)

This is dependency injection, and it is the single highest-leverage decision you can make for testability. Once Summarizer accepts an llm_client object rather than constructing one internally, you can pass in anything that has a .complete() method, including a fake.

Pattern 1: hand-rolled fakes

The simplest mock is a plain object that mimics the interface.

class FakeLLMClient:
    def __init__(self, canned_response: str):
        self.canned_response = canned_response
        self.calls = []

    def complete(self, prompt: str, max_tokens: int = 1024) -> str:
        self.calls.append(prompt)
        return self.canned_response


def test_summarizer_calls_llm_with_correct_prompt():
    fake = FakeLLMClient(canned_response="A short summary.")
    summarizer = Summarizer(fake)

    result = summarizer.summarize("Some long article text here.")

    assert result == "A short summary."
    assert "Some long article text here." in fake.calls[0]

This test runs in microseconds, has zero network dependency, and clearly documents what your code expects from the LLM boundary. The calls list also lets you assert on prompt construction, which matters more than people think: a bug where you forgot to include the user's actual input in the prompt will not show up if you only check the return value.

Hand-rolled fakes work well when the interface is small and stable. As soon as you need different responses for different inputs, or you need to simulate errors, reach for something more flexible.

class ScriptedLLMClient:
    def __init__(self, responses: list[str]):
        self.responses = list(responses)
        self.calls = []

    def complete(self, prompt: str, max_tokens: int = 1024) -> str:
        self.calls.append(prompt)
        if not self.responses:
            raise RuntimeError("No more scripted responses")
        return self.responses.pop(0)


def test_agent_retries_once_on_bad_json():
    scripted = ScriptedLLMClient(responses=["not json", '{"status": "ok"}'])
    agent = JsonAgent(scripted)

    result = agent.run("do the thing")

    assert result["status"] == "ok"
    assert len(scripted.calls) == 2

This pattern is great for testing retry logic, multi-turn agent loops, and tool-use chains where the response sequence matters.

Pattern 2: mocking with `unittest.mock` or `pytest-mock`

When you cannot (or do not want to) inject a fake client, you can patch the SDK call directly. This is the standard approach in Python, and most other languages have an equivalent (Jest's jest.mock in JavaScript, unittest::mock crates in Rust, and so on).

from unittest.mock import patch, MagicMock

def test_summarizer_with_patched_client():
    mock_response = MagicMock()
    mock_response.content = [MagicMock(text="Mocked summary.")]

    with patch("anthropic.Anthropic") as mock_anthropic_class:
        mock_instance = mock_anthropic_class.return_value
        mock_instance.messages.create.return_value = mock_response

        client = LLMClient(api_key="fake-key")
        result = client.complete("Summarize this.")

        assert result == "Mocked summary."
        mock_instance.messages.create.assert_called_once()

Using pytest-mock (the mocker fixture) tidies this up and handles cleanup automatically:

def test_summarizer_with_pytest_mock(mocker):
    mock_create = mocker.patch("anthropic.resources.messages.Messages.create")
    mock_create.return_value.content = [mocker.MagicMock(text="Mocked summary.")]

    client = LLMClient(api_key="fake-key")
    result = client.complete("Summarize this.")

    assert result == "Mocked summary."

Patching is more powerful because you do not need to change your production code's constructor signatures, but it is also more brittle. If the SDK changes its internal method names or response shape (which happens more often than you would like with fast-moving provider SDKs), your patches break even though your actual behavior is fine. Prefer dependency injection with a thin wrapper (Pattern 1) when you control the code, and reserve patching for third-party code you cannot restructure.

Pattern 3: HTTP-level mocking with `responses` or `httpx` mock transports

Sometimes you want to test closer to the wire, verifying that your code sends the right HTTP request and correctly parses a realistic response body, including headers, streaming chunks, and error status codes. For this, mock at the HTTP layer instead of the SDK layer.

import responses
import json

@responses.activate
def test_llm_call_hits_correct_endpoint():
    responses.add(
        responses.POST,
        "https://api.anthropic.com/v1/messages",
        json={
            "id": "msg_test123",
            "type": "message",
            "role": "assistant",
            "content": [{"type": "text", "text": "Hello from the mock."}],
            "model": "claude-sonnet-5",
            "stop_reason": "end_turn",
            "usage": {"input_tokens": 10, "output_tokens": 5},
        },
        status=200,
    )

    client = LLMClient(api_key="fake-key")
    result = client.complete("Say hello.")

    assert result == "Hello from the mock."
    assert len(responses.calls) == 1
    sent_body = json.loads(responses.calls[0].request.body)
    assert sent_body["model"] == "claude-sonnet-5"

This approach is useful when you are writing your own thin HTTP wrapper instead of using the official SDK, or when you specifically want to test error handling for rate limits and server errors.

@responses.activate
def test_llm_call_handles_rate_limit():
    responses.add(
        responses.POST,
        "https://api.anthropic.com/v1/messages",
        json={"type": "error", "error": {"type": "rate_limit_error", "message": "Rate limited"}},
        status=429,
    )

    client = LLMClient(api_key="fake-key")

    with pytest.raises(RateLimitError):
        client.complete("Say hello.")

Simulating 429s, 500s, and timeouts is one of the most valuable things HTTP-level mocking gives you, because that is exactly the code path most teams forget to test until it fails in production during a provider outage.

Pattern 4: record and replay with VCR-style cassettes

Hand-written mocks are great for unit tests, but they drift from reality: you write {"status": "ok"} by hand and it stops matching what the real API actually returns after a schema change. Record-and-replay tools solve this by capturing a real API response once, saving it to a file (a "cassette"), and replaying it on every subsequent test run.

In Python, vcrpy (or pytest-recording, which wraps it) is the standard tool.

import pytest

@pytest.mark.vcr()
def test_summarizer_integration():
    client = LLMClient(api_key="fake-key-not-used-on-replay")
    summarizer = Summarizer(client)

    result = summarizer.summarize("The quick brown fox jumps over the lazy dog.")

    assert "fox" in result.lower() or len(result) > 0

The first time this test runs (with --record-mode=once and a real API key set), it makes a genuine API call and saves the request and response to a YAML file like cassettes/test_summarizer_integration.yaml. Every run after that replays the saved response instead of hitting the network, so the test is fast and free, but the response shape is guaranteed to match what the real API actually sends.

This pattern is the best of both worlds for integration-style tests: realistic payloads, zero network dependency in CI, and an explicit, reviewable diff whenever you re-record a cassette after a genuine behavior change. The tradeoff is that cassettes can go stale (the real API evolves but your saved response does not), so re-record periodically, and never commit a cassette containing a real API key. Most tools scrub the Authorization header automatically, but check the output before committing.

Testing streaming responses

Streaming is where a lot of mocking setups fall apart, because the interface is an iterator of chunks rather than a single return value. Mock the chunk sequence explicitly.

class FakeStreamingLLMClient:
    def __init__(self, chunks: list[str]):
        self.chunks = chunks

    def stream(self, prompt: str):
        for chunk in self.chunks:
            yield chunk


def test_streaming_consumer_assembles_full_text():
    fake = FakeStreamingLLMClient(chunks=["Hel", "lo, ", "world", "!"])
    consumer = StreamingConsumer(fake)

    full_text = consumer.consume("irrelevant prompt")

    assert full_text == "Hello, world!"

If you use the real SDK's streaming interface, patch it to return an iterable of mock event objects that match the shape your parsing code expects (content_block_delta events, a final message_stop, and so on). Write one test that checks you handle a clean stream, and one that checks you handle a stream that ends abruptly mid-response, since that is a real failure mode with network interruptions.

Testing tool use and multi-step agent loops

When your code drives an agent loop (call the model, execute a tool call it requests, feed the result back), mock at the level of "one LLM turn returns one of these three things": a text response, a tool-use request, or an error. A scripted fake client works well here because the sequence of responses is the whole point of the test.

def test_agent_executes_tool_then_returns_final_answer():
    scripted = ScriptedLLMClient(responses=[
        json.dumps({"tool": "search", "args": {"query": "weather in Delhi"}}),
        "The weather in Delhi is sunny.",
    ])
    agent = ToolAgent(scripted, tools={"search": lambda args: "sunny, 32C"})

    result = agent.run("What's the weather in Delhi?")

    assert "sunny" in result.lower()
    assert len(scripted.calls) == 2

This lets you verify the full loop, tool selection, tool execution, and final synthesis, without ever touching a real model or a real search API.

Structuring your test suite: unit, integration, and live tiers

A healthy setup for LLM-touching code usually has three tiers.

• Unit tests: fully mocked, no network, run on every commit. Use hand-rolled fakes or unittest.mock patches. These should be the bulk of your suite and run in seconds.
• Integration tests: use VCR-style cassettes so payload shapes stay realistic, still no live network in normal CI runs. Re-record cassettes deliberately when you change providers or models.
• Live/smoke tests: a small number of tests that actually call the real API, gated behind an environment variable or a separate CI job that runs on a schedule (nightly, say) rather than on every push. These catch real model drift, deprecated model names, and genuine API contract changes.

import os
import pytest

live = pytest.mark.skipif(
    os.environ.get("RUN_LIVE_LLM_TESTS") != "1",
    reason="Live LLM tests only run when RUN_LIVE_LLM_TESTS=1",
)

@live
def test_real_api_returns_reasonable_summary():
    client = LLMClient(api_key=os.environ["ANTHROPIC_API_KEY"])
    summarizer = Summarizer(client)
    result = summarizer.summarize("A long article about renewable energy...")
    assert len(result) > 0

Keep this tier small. Its job is to catch drift, not to re-verify logic your mocked unit tests already cover.

Common mistakes to avoid

Mocking too close to the model's actual reasoning. If your test hardcodes "The answer is 42" and asserts the code returns exactly that, you are testing your mock, not your code. Assert on structure and behavior (did it parse correctly, did it call the right tool, did it retry) rather than exact model wording.

Forgetting to assert on the prompt. Mocking the response is only half the job. Many real bugs live in prompt construction: a missing variable, a wrong system message, an omitted piece of context. Capture and assert on what was actually sent to the model, not just what came back.

Never testing error paths. Rate limits, timeouts, malformed JSON from the model, empty responses, content filtering refusals, these all happen in production. If your test suite only has the happy path mocked, you will find out about these in an incident, not in CI.

Letting cassettes rot. Recorded fixtures from six months ago may no longer match the current API version or model behavior. Add a periodic task (monthly, or tied to model upgrades) to re-record integration cassettes against the live API.

Skipping mocking for "just a quick script." Scripts turn into production code more often than anyone plans. If a script calls an LLM and has any logic worth getting right, it deserves the same seam-and-mock treatment as application code.

FAQ

Do I need a different mocking approach for every LLM provider? No. If you wrap each provider behind your own thin client interface (a complete() or chat() method), your business logic and its tests never need to know which provider is underneath. You only need provider-specific mocking at the boundary layer itself, which should be a small, well-tested piece of code.

Should I mock the SDK or the raw HTTP layer? Mock the SDK (or better, your own wrapper around it) for most unit tests, since it is faster to write and easier to read. Drop to HTTP-level mocking with tools like responses or an httpx mock transport when you specifically need to verify request headers, exact JSON payloads, or error status code handling.

How do I avoid committing API keys in recorded cassettes? Use a VCR tool's built-in filtering (filter_headers=["authorization"] in vcrpy) to scrub sensitive headers before the cassette is written to disk. Also add a pre-commit check or CI grep that fails the build if a cassette file contains anything resembling a real key pattern.

Is it worth testing against a live model at all? Yes, but sparingly. A small, separate suite of live tests catches things mocks cannot: deprecated model names, subtle prompt regressions, or provider-side behavior changes. Run these on a schedule or before releases, not on every commit, so your main CI stays fast and free.

What's the fastest way to start if my codebase has zero seams around LLM calls today? Pick the single most-tested piece of LLM-touching logic, wrap the provider call in a one-method class, and inject it. You do not need to refactor everything at once. Every new feature you write after that should go through the wrapper, and the untested legacy calls can be migrated opportunistically.