Contextual Retrieval: Anthropic's Technique for Better RAG Chunks

The Chunk That Forgot Where It Came From

Picture a chunk from an SEC filing that reads: "The company's revenue grew by 3% over the previous quarter." On its own, that sentence is nearly useless. Which company? Which quarter? Is 3% growth good or bad for this business? A human reading the full filing would never lose that context, but a RAG (Retrieval-Augmented Generation) pipeline splits documents into chunks before anyone gets to ask a question, and once you split, the surrounding context is gone. The embedding model sees only the fragment, and it embeds the fragment's words, not the meaning that depended on the paragraph three pages earlier.

This is not a minor edge case. It is the core failure mode of naive chunking, and it is why so many RAG systems retrieve confidently wrong chunks or miss the right one entirely. Anthropic's Contextual Retrieval, published in 2024, is a direct, practical fix for this problem. It does not require a new vector database, a new embedding model, or a fundamentally different retrieval architecture. It changes one thing: what you embed and index in the first place. If you are building or debugging a RAG system, this is one of the highest-leverage changes you can make, and it is the kind of technique we walk through hands-on in our Introduction to RAG course at teachyou.ai.

Why Standard Chunking Breaks Retrieval

To understand why Contextual Retrieval works, you need to understand exactly how standard RAG chunking fails.

A typical RAG pipeline does the following:

1. Take a large document (a manual, a codebase, a legal contract, a knowledge base article).
2. Split it into chunks of some fixed size, say 200-800 tokens, sometimes with overlap.
3. Embed each chunk independently using an embedding model.
4. Store the embeddings in a vector database, often alongside a BM25 or keyword index.
5. At query time, embed the user's question and retrieve the top-k most similar chunks.
6. Feed those chunks to an LLM as context for generating an answer.

The failure happens at step 3. Each chunk is embedded independently, with zero awareness of the document it came from. A chunk that says "the API returns a 429 status code" carries no information about which API, which endpoint, or which version of the service. A chunk that says "this clause does not apply to Tier 2 customers" doesn't say which clause or which contract. The embedding model does its best with the words it's given, but the words it's given are incomplete.

This shows up in two concrete ways:

• Semantic retrieval misses. The embedding vector for a context-free chunk drifts away from the vector a user's query would produce, because the query usually includes the missing context ("What is ACME Corp's Q2 2023 revenue growth?") while the chunk does not.
• Lexical retrieval misses. BM25 and other keyword-based methods fail even harder, because they rely on exact term overlap. If the chunk never mentions "ACME Corp" or "Q2 2023," a BM25 search for those terms will not surface it, no matter how relevant the chunk actually is.

Most teams respond to this by throwing more engineering at the margins: bigger overlap windows, hybrid search, more aggressive reranking, larger k. Contextual Retrieval instead attacks the root cause: the chunk itself is missing information, so give the chunk that information back before you ever embed or index it.

Think about how this plays out in a real support-docs scenario. Suppose you're building a RAG system over a product's documentation, and a user asks, "Does the free tier support webhook retries?" The correct chunk might be a single bullet point buried in a "Limitations" section three levels deep in a page about the Pro tier. That bullet point, in isolation, might just say "Retries are capped at 3 attempts." It doesn't say "free tier," it doesn't say "webhooks," and it doesn't even say "Pro tier" — all of that context lives in the page title and the section heading above it, which standard chunking usually throws away. No amount of clever reranking can save you here, because reranking only re-scores the chunks retrieval already surfaced. If the chunk never entered the candidate set in the first place because its embedding and its keywords didn't match the query, reranking never gets a chance to fix that.

The Core Idea: Prepend Context Before You Embed

The mechanism is disarmingly simple. Before embedding or indexing a chunk, you generate a short piece of context — typically 50 to 100 tokens — that situates the chunk within the full document, and you prepend that context directly to the chunk text.

Take the earlier example. The original chunk:

The company's revenue grew by 3% over the previous quarter.

After Contextual Retrieval, the chunk that actually gets embedded and indexed looks like this:

This chunk is from an SEC filing on ACME corp's performance in Q2 2023;
the previous quarter's revenue was $314 million. The company's revenue
grew by 3% over the previous quarter.

Notice what changed and what didn't. The original sentence is untouched — you have not lost or altered any information. You've added a short preamble that answers the questions a reader would otherwise have to infer from surrounding pages: which company, which filing, which quarter, and even a relevant number from earlier in the document (the previous quarter's revenue) that helps situate the 3% figure.

Now when this augmented chunk gets embedded, the resulting vector actually reflects "ACME Corp, Q2 2023, revenue growth" instead of just "revenue grew 3%, previous quarter." A query like "How did ACME's revenue change in Q2 2023?" is now much closer in embedding space to this chunk, because the chunk's text finally contains the same anchoring details the query does. And critically, this same augmented text also gets indexed for BM25 or any other keyword-based method, so exact-match search on "ACME Corp" or "Q2 2023" now works too.

This is why Anthropic calls it Contextual Retrieval rather than just "contextual embeddings" — the technique deliberately improves both the semantic (embedding) side and the lexical (keyword) side of a hybrid retrieval system at the same time, using the same augmented chunk text.

Generating Context Automatically With an LLM

Writing a custom context sentence for every chunk in a large corpus by hand is not realistic. The insight that makes this technique practical is that you can have an LLM generate the context automatically, chunk by chunk, using the full document as reference.

The approach: for each chunk, send the model the entire source document plus that specific chunk, and ask it to produce a short, specific description that would help situate the chunk for search purposes. The prompt is intentionally plain — something along the lines of:

<document>
{{WHOLE_DOCUMENT}}
</document>

Here is the chunk we want to situate within the whole document:
<chunk>
{{CHUNK_CONTENT}}
</chunk>

Please give a short, succinct context to situate this chunk within
the overall document for the purposes of improving search retrieval
of the chunk. Answer only with the succinct context and nothing else.

Run this once per chunk across your entire corpus, and you get a context-augmented version of every chunk, ready to embed and index. The resulting context is short by design — Anthropic reports typical outputs of 50 to 100 tokens — because the goal is a compact anchor, not a summary of the whole document.

A natural objection here is cost: doesn't sending the whole document for every single chunk get expensive fast, especially for documents with hundreds of chunks? This is where prompt caching becomes essential, not optional.

Why Prompt Caching Makes This Affordable

If you naively re-send the full document as part of the prompt for every chunk, you pay for the full input tokens of that document over and over, once per chunk. For a document with 100 chunks, you'd be paying for the document's token count 100 times over. That cost profile would make Contextual Retrieval impractical for any large corpus.

Prompt caching solves this. With Claude's prompt caching, you load the reference document into the cache once, and then for each chunk you only pay the (much smaller) cost of the chunk-specific portion of the prompt plus a steep discount on the cached document tokens for every subsequent call. Anthropic's own numbers put the resulting cost at roughly $1.02 per million tokens of source document when generating context for every chunk in that document. That is a one-time preprocessing cost, not a per-query cost, and it makes contextualizing an entire knowledge base a genuinely practical option rather than a research curiosity.

The workflow, concretely, looks like this:

1. Load the full document into the model's context once, marked for caching.
2. For each chunk derived from that document, send a short request referencing the cached document and asking only for that chunk's situating context.
3. Prepend the returned context to the chunk.
4. Embed the augmented chunk and add it to your vector index.
5. Also add the augmented chunk to your BM25 / keyword index.

This is a preprocessing pipeline you run once when ingesting or re-ingesting documents, not something that happens at query time. Query-time latency is unaffected; you're paying a small, cacheable, one-time cost up front in exchange for meaningfully better retrieval on every future query against that corpus.

A Minimal Working Example

Here's a simplified Python sketch of the ingestion pipeline, stripped down to the essential logic so you can see how the pieces fit together:

import anthropic

client = anthropic.Anthropic()

CONTEXT_PROMPT = """<document>
{document}
</document>

Here is the chunk we want to situate within the whole document:
<chunk>
{chunk}
</chunk>

Please give a short, succinct context to situate this chunk within
the overall document for the purposes of improving search retrieval
of the chunk. Answer only with the succinct context and nothing else.
"""

def generate_chunk_context(document: str, chunk: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=200,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": CONTEXT_PROMPT.format(document=document, chunk=chunk),
                        "cache_control": {"type": "ephemeral"},
                    }
                ],
            }
        ],
    )
    return response.content[0].text.strip()


def build_contextual_chunks(document: str, raw_chunks: list[str]) -> list[str]:
    contextual_chunks = []
    for chunk in raw_chunks:
        context = generate_chunk_context(document, chunk)
        contextual_chunks.append(f"{context}\n\n{chunk}")
    return contextual_chunks


# Ingestion pipeline
raw_chunks = split_into_chunks(document_text)          # your existing chunker
contextual_chunks = build_contextual_chunks(document_text, raw_chunks)

for chunk_text in contextual_chunks:
    embedding = embed(chunk_text)                       # your embedding model
    vector_store.upsert(chunk_text, embedding)
    bm25_index.add(chunk_text)                           # your keyword index

A few implementation notes worth calling out:

• The cache_control marker on the document portion of the prompt is what makes repeated calls with the same document cheap. Structure your prompt so the large, unchanging document comes first and the small, per-chunk content comes last, since caching works on prefix matches.
• You still keep your existing chunking logic (split_into_chunks). Contextual Retrieval doesn't replace how you split documents — it augments what you do with each chunk afterward.
• Both the vector store and the keyword index receive the same contextualized text, which is what lets this technique improve semantic and lexical retrieval simultaneously.

What the Numbers Actually Show

It's worth being precise about the evaluation results here, because the size of the improvement is what makes this worth adopting rather than filing away as a nice-to-have.

Anthropic measured retrieval performance using top-20-chunk retrieval failure rate — essentially, how often the correct chunk fails to appear anywhere in the top 20 results returned by the retrieval system. Lower is better. The baseline failure rate with standard, non-contextual embeddings and BM25 was 5.7%.

• Contextual Embeddings alone (just the embedding side, augmented) reduced the failure rate to 3.7%, a 35% relative reduction.
• Contextual Embeddings plus Contextual BM25 (both semantic and lexical retrieval augmented) brought the failure rate down to 2.9%, a 49% relative reduction.
• Adding a reranking step on top of both pushed the failure rate down to 1.9%, a 67% relative reduction from baseline.

The pattern here matters as much as the headline numbers: the biggest single jump comes from contextualizing the embeddings, but combining contextual embeddings with contextual BM25 captures gains that neither one gets alone, because they catch different kinds of queries — semantic similarity for conceptual questions, exact-match for specific terms, names, and numbers. Reranking then squeezes out further improvement by re-scoring the already-improved candidate set with a more expensive, more accurate model at query time. Each layer is complementary rather than redundant, which is exactly the kind of stacked-improvement architecture worth understanding if you're designing a production retrieval pipeline rather than a demo.

Practical Considerations Before You Adopt This

Contextual Retrieval is simple in concept but there are real decisions to make when you put it into practice.

• Chunk boundaries still matter. Contextual Retrieval fixes the "missing context" problem, but it doesn't fix badly-drawn chunk boundaries. If your chunker is splitting mid-sentence or separating a table from its caption, you'll still have problems. Get your chunking strategy reasonably sound first, then layer contextualization on top.
• Embedding model choice still matters. Anthropic's evaluation found that stronger embedding models (they specifically tested and favored Gemini and Voyage embedding models) benefited from and amplified the gains from contextualization. Contextual Retrieval is an enhancement to your retrieval stack, not a replacement for choosing a good embedding model.
• Tune the number of retrieved chunks. The evaluation found that retrieving around 20 chunks worked meaningfully better than retrieving 5 or 10. If you've hardcoded a small top_k because that's what fit your context window, revisit that assumption — with contextualized chunks and a reranking step, you can afford to cast a wider net at the retrieval stage and let reranking narrow it down.
• Customize the context prompt for your domain. The generic "situate this chunk" prompt works well as a default, but if you're working with codebases, legal contracts, or medical literature, a domain-tuned prompt (mentioning function names, clause numbers, or terminology conventions) will produce sharper, more useful context.
• This is a preprocessing cost, run it accordingly. Because context generation happens once per chunk at ingestion time, it's a good candidate for batch processing pipelines. Anthropic's own API also offers a Batches API for exactly this kind of large, non-latency-sensitive workload, which can reduce costs further on top of prompt caching.
• Always evaluate on your own data. The 35% / 49% / 67% figures come from Anthropic's evaluation corpus. Your documents, your query patterns, and your existing retrieval setup will produce different numbers. Build a small evaluation set of realistic queries with known-correct chunks before and after adopting this technique, so you can measure the actual lift rather than assuming it transfers exactly.

Where This Fits in a Real RAG Pipeline

It's worth zooming out and placing Contextual Retrieval within the broader RAG pipeline, because it is not a standalone system — it's a preprocessing enhancement that slots into a pipeline you likely already have.

A production RAG system generally has three distinct stages: ingestion (chunking, embedding, indexing), retrieval (semantic search, keyword search, fusion, reranking), and generation (feeding retrieved chunks to an LLM to produce an answer). Contextual Retrieval lives entirely in the ingestion stage. It changes the text you chunk, embed, and index, but it does not require you to change your vector database, your retrieval algorithm, or your generation prompt. This is precisely why it's such a good return on engineering effort: you can add it to an existing RAG system without a rewrite, measure the retrieval quality improvement directly, and decide whether the added ingestion cost (which, thanks to prompt caching, is modest) is worth it for your use case.

It also composes cleanly with other RAG improvements. Hybrid search (combining semantic and keyword retrieval) becomes more powerful once both sides are contextualized. Reranking becomes more effective when the candidate set it's re-scoring is already higher quality. Query rewriting and HyDE-style techniques on the query side pair naturally with better chunk representations on the document side. None of these techniques compete with Contextual Retrieval — they stack.

There's also a maintenance dimension worth planning for. Documents change. When a source document is updated, the chunks derived from it need to be re-contextualized, not just re-embedded, because the situating context for a chunk depends on the surrounding document state at the time it was generated. A practical pattern is to treat contextualization as part of your document versioning pipeline: whenever a document is re-ingested, regenerate context for every chunk derived from it rather than trying to patch individual chunks. Because of prompt caching, re-running the full contextualization pass on an updated document is still cheap relative to the value of keeping your index accurate, so there's little reason to skip this step even for frequently-updated content like living documentation or policy pages.

One more thing worth flagging for teams evaluating this: Contextual Retrieval is not exclusive to any one vector database or search stack. Because the technique only changes the text that gets embedded and indexed, it works whether you're using a managed vector database, a self-hosted one, or a simple in-memory index for a smaller project. The same is true of the keyword side — whether you're running Elasticsearch, a lightweight BM25 library, or a Postgres full-text search column, the contextualized chunk text drops in the same way. This portability is part of why the technique has been picked up quickly across different RAG stacks since it was published: it doesn't ask you to standardize on a particular vendor or framework, it just asks you to change what you feed into the tools you already have.

Closing Thoughts

The uncomfortable truth about most RAG systems in production is that they fail silently. The system doesn't crash; it just quietly retrieves the wrong chunk, or fails to retrieve the right one, and the LLM confidently generates an answer from incomplete information. Contextual Retrieval is valuable precisely because it targets that silent failure mode directly, at the point where it originates: the moment a chunk is stripped of the document context that gave it meaning.

What makes this technique worth learning in depth rather than skimming as a blog post is that it's a genuine engineering pattern, not a one-off trick. Using an LLM to enrich data before it enters a retrieval index, paying for that enrichment efficiently via caching, and evaluating the result with a proper failure-rate metric rather than vibes — that combination of ideas shows up again and again in serious retrieval and search systems, well beyond this one specific application.

If you want to go deeper — building a full RAG pipeline from raw documents through chunking, embedding, contextualization, hybrid retrieval, reranking, and generation, with hands-on exercises rather than just theory — that's exactly what we cover in Introduction to RAG here at teachyou.ai.