Contextual Compression in RAG Pipelines: A Practical Guide

Contextual compression in RAG is the step where you take the chunks your retriever pulled back and filter or rewrite them down to only the content relevant to the current query, before they ever reach the generator. It sits between retrieval and generation, and it exists because "relevant enough to retrieve" and "relevant enough to answer with" are not the same bar. A chunk can score well on cosine similarity and still be 80% filler that pushes your real answer out of the model's attention. This guide walks through the mechanics, the code, and the tradeoffs of adding a compression stage to a retrieval augmented generation pipeline.

What Contextual Compression Actually Solves

Standard RAG retrieves fixed-size chunks, typically 200 to 1000 tokens, because that is how the documents were split at ingestion time. Chunk boundaries do not know where the useful sentence starts or ends. A chunk about "refund policy" might contain one sentence that answers "how long do refunds take" and four sentences about eligibility criteria that are irrelevant to that specific question. When you stack five or ten of these chunks into a prompt, the signal-to-noise ratio drops fast.

Three concrete problems follow from this:

• Context window pressure. Even with generous context windows, every irrelevant token you send is a token the model has to read, weigh, and discard. That costs latency and money on every single call.
• Lost-in-the-middle effects. Models are measurably less reliable at using information buried in the middle of a long context versus the start or end. Padding a prompt with noise increases the odds the answer-bearing sentence lands in the dead zone.
• Hallucination surface area. More irrelevant text gives the model more opportunity to synthesize a plausible-sounding but wrong answer by blending unrelated facts.

Contextual compression in RAG pipelines addresses all three by shrinking what gets passed to the generator without changing what gets retrieved. You still cast a wide net at retrieval time, you just don't hand the model the whole net.

Where Compression Fits in the Pipeline

A typical RAG flow looks like: query -> embed -> vector search -> top-k chunks -> generation. Contextual compression inserts one stage:

query -> embed -> vector search -> top-k chunks -> compress -> generation

The compressor takes the query and each retrieved chunk as input, and returns either a trimmed version of the chunk, a relevance score used to drop the chunk entirely, or both. This is distinct from reranking, which reorders the full chunk list without touching chunk content, and distinct from chunking strategy, which is a decision made once at ingestion time. Compression is a per-query, per-chunk operation that happens at retrieval time, every time.

Three Compression Techniques, in Order of Cost

1. Extraction (cheapest, fastest). An LLM or a lightweight extractive model reads the chunk and the query, then returns only the sentences from the chunk that are relevant, verbatim, with no rewriting. This is the safest option because it cannot introduce facts that were not already in the source text. It is a good default for regulated or high-stakes domains where every word in the answer needs to be traceable back to a source string.

2. Filtering (cheapest of all, binary). Instead of trimming text, filtering just decides keep or discard for each chunk based on a relevance score against the query. This is often done with a cross-encoder or a small classifier rather than a full LLM call, so it is fast enough to run over every candidate. Filtering does not reduce noise within a kept chunk, it only removes entire chunks that would not have helped anyway.

3. Summarization (most expensive, most lossy). An LLM rewrites the chunk into a shorter form that preserves the query-relevant content in the fewest possible tokens. This gets you the smallest final prompt, but introduces paraphrase risk: the summary might drop a caveat, a number, or a condition that mattered. Reserve this for chunks that are large and mostly irrelevant, where extraction alone would not save enough tokens.

Most production pipelines combine filtering (drop the clearly irrelevant chunks first) with extraction (trim what remains) and skip summarization unless token budgets are extremely tight.

Building a Contextual Compression Retriever

Below is a self-contained implementation that does not depend on a specific framework's compression API, since those change shape often. It uses your existing LLM client directly, so you can swap in any provider.

import json
from dataclasses import dataclass


@dataclass
class Chunk:
    text: str
    source: str
    score: float


def extract_relevant_sentences(llm_client, query: str, chunk: Chunk) -> str | None:
    """Ask the model to pull only the sentences relevant to the query.
    Returns None if nothing in the chunk is relevant."""
    prompt = f"""Given the query and the document excerpt below, extract ONLY
the sentences from the excerpt that are directly relevant to answering
the query. Copy them verbatim, do not paraphrase. If nothing is relevant,
respond with exactly: NONE

Query: {query}

Excerpt:
{chunk.text}

Relevant sentences:"""

    response = llm_client.complete(prompt, max_tokens=300, temperature=0)
    result = response.strip()
    if result == "NONE" or not result:
        return None
    return result


def compress_chunks(llm_client, query: str, chunks: list[Chunk]) -> list[Chunk]:
    compressed = []
    for chunk in chunks:
        trimmed_text = extract_relevant_sentences(llm_client, query, chunk)
        if trimmed_text is not None:
            compressed.append(
                Chunk(text=trimmed_text, source=chunk.source, score=chunk.score)
            )
    return compressed

This is a working baseline, but it makes one LLM call per chunk, which is slow if you retrieve ten or twenty candidates per query. In practice you batch this.

Batching Compression to Cut Latency

Instead of one call per chunk, send all candidate chunks in a single request and ask the model to return a structured decision for each one. This trades a slightly harder parsing step for a five-to-tenfold reduction in round trips.

def batch_compress(llm_client, query: str, chunks: list[Chunk]) -> list[Chunk]:
    numbered = "\n\n".join(
        f"[{i}] {c.text}" for i, c in enumerate(chunks)
    )

    prompt = f"""You will see a query and several numbered document excerpts.
For each excerpt, extract only the sentences relevant to the query.
If an excerpt has nothing relevant, omit it from the output.

Return a JSON array of objects with fields "index" and "relevant_text".
Do not include any excerpt that has no relevant content.

Query: {query}

Excerpts:
{numbered}

JSON output:"""

    response = llm_client.complete(prompt, max_tokens=1200, temperature=0)

    try:
        results = json.loads(response)
    except json.JSONDecodeError:
        # Fall back to returning chunks uncompressed rather than dropping
        # everything on a parse failure.
        return chunks

    compressed = []
    for item in results:
        idx = item["index"]
        if 0 <= idx < len(chunks):
            original = chunks[idx]
            compressed.append(
                Chunk(
                    text=item["relevant_text"],
                    source=original.source,
                    score=original.score,
                )
            )
    return compressed

The fallback on parse failure matters. A compression stage that silently returns an empty context on a malformed JSON response is worse than no compression at all, because the generator will confidently answer from nothing. Always fail open to the uncompressed chunks, never fail closed to an empty context.

Filtering First, Then Extracting

Running extraction on every retrieved chunk wastes calls on chunks that should have been dropped entirely. A cheaper filter pass first narrows the field:

def filter_by_relevance(cross_encoder, query: str, chunks: list[Chunk], threshold: float = 0.3) -> list[Chunk]:
    pairs = [(query, c.text) for c in chunks]
    scores = cross_encoder.predict(pairs)
    return [c for c, s in zip(chunks, scores) if s >= threshold]


def compression_pipeline(cross_encoder, llm_client, query: str, chunks: list[Chunk]) -> list[Chunk]:
    filtered = filter_by_relevance(cross_encoder, query, chunks)
    if not filtered:
        # nothing cleared the bar, fall back to the original top chunks
        # rather than returning an empty context
        filtered = chunks[:3]
    return batch_compress(llm_client, query, filtered)

A local cross-encoder runs in milliseconds and does not touch your LLM API budget, so it is worth the extra dependency if you are compressing at any real query volume. Sentence-transformers style cross-encoder models are the standard choice here and run fine on CPU for typical chunk counts.

Chunk-Level vs Document-Level Compression

Everything above compresses at the chunk level: each retrieved chunk is independently trimmed. There is a second mode worth knowing about, document-level compression, where you first group chunks by source document, then compress the concatenated group as a unit. This matters when a single document contributes multiple chunks that only make sense together, for example a chunk with a table header and a separate chunk with the table rows. Compressing them independently can strip context that was only implicit across the chunk boundary.

The tradeoff is that document-level compression means larger inputs per compression call, so it costs more tokens and more latency. Use it selectively, for document types where your chunking strategy is known to split related content (long tables, numbered procedures, multi-step code examples), and use chunk-level compression as the default everywhere else.

Choosing a Compressor Model

You do not need your largest, most expensive model to run extraction. Extraction is a narrow, well-defined task: find the relevant sentences and copy them. A smaller, faster model in the same family as your generator, or a distilled model trained for extraction, usually performs the task at high enough accuracy while keeping the added latency small relative to the generation call it precedes.

A rule of thumb: if compression adds more latency than it saves in generation tokens, the pipeline is misconfigured. Profile end-to-end request time with and without compression before shipping it, not just token counts, since the extra LLM round trip has its own fixed overhead.

For the filtering stage, a cross-encoder reranker model is almost always the right tool over an LLM call, because it is purpose-built for the pairwise relevance task and runs orders of magnitude faster.

Measuring Whether Compression Is Helping

Do not add a compression stage on faith. Instrument these before and after:

• Context token count per request. The most direct measure of whether compression is doing its job. Track median and p95, since compression benefits vary a lot by query type.
• Answer groundedness. Sample generated answers and check whether every claim traces back to the compressed context. If groundedness drops after adding compression, your extraction step is likely stripping caveats or qualifying clauses along with the noise.
• End-to-end latency. Compression adds a hop. Measure total request time, not just generation time, to see if the token savings on generation outweigh the extraction call's own cost.
• Answer recall on a held-out eval set. Run a fixed set of query-answer pairs through the pipeline with compression on and off. If recall drops, your compressor is discarding sentences that were actually load-bearing, usually because the extraction prompt is too aggressive or the threshold on the filter stage is too high.

Keep the eval set small enough to run on every pipeline change but large enough to catch regressions, twenty to fifty representative query-answer pairs is a reasonable starting point for most teams.

Common Failure Modes

Over-aggressive extraction drops qualifiers. An extraction prompt that only looks for "sentences that answer the query" will sometimes drop a sentence like "this only applies to accounts opened after 2024" because it does not look like a direct answer, even though it is essential context for correctness. Fix this by explicitly instructing the extractor to keep conditions, exceptions, and caveats attached to any sentence it extracts.

Compression breaks structured content. If a chunk contains a code block, a table, or a numbered list, sentence-level extraction can mangle it, pulling out a table row without its header or a code line without the surrounding function. Detect structured content before compressing (a simple check for fenced code blocks, markdown tables, or numbered list markers) and skip compression for those chunks, passing them through whole.

Silent empty context on parse failures. Covered above, but worth repeating: any JSON-based batch compression call needs a fallback path. A malformed response should never resolve to "send nothing to the generator."

Compression cost exceeds the savings. If your chunks are already small and well-targeted from ingestion-time chunking, adding a compression stage before every generation call is pure overhead. Compression pays off when retrieval routinely returns chunks that are meaningfully larger than what a single answer needs, not when chunks are already tight.

Query-chunk mismatch on multi-turn conversations. In a chat interface, the "query" for compression purposes should usually be a reformulated, context-aware version of the user's latest message, not the raw message. "What about last year?" compressed against the literal string "what about last year" will not extract anything useful. Rewrite the query with conversation context before it reaches the compression stage.

Contextual Compression vs Reranking: When to Use Which

These two stages are often confused because both sit between retrieval and generation and both use a query-chunk relevance signal.

Reranking reorders the full list of retrieved chunks by a more accurate relevance score than the original vector search similarity, then truncates to a smaller top-k. It does not change chunk content. Reranking is cheap, fast, and should be close to a default in any RAG pipeline that retrieves more than five or so candidates, since first-pass vector search similarity is a rough signal.

Contextual compression changes the content of the chunks that make it through, either by trimming or filtering. It costs more (an LLM call per chunk or batch, versus a lightweight scoring pass for reranking) and is not always necessary.

A practical pipeline order: retrieve a generous top-k (twenty to fifty candidates), rerank down to a smaller set (five to ten), then compress that smaller set before generation. Reranking does the coarse cut cheaply, compression does the fine cut on the survivors. Running compression on fifty raw candidates instead of ten reranked ones wastes calls on chunks that reranking would have dropped anyway.

Production Checklist

• Rerank before you compress, not instead of it. They solve different problems.
• Always fail open: a parse failure or empty compression result should return the original chunk, never an empty context.
• Skip compression for chunks containing code blocks, tables, or other structured content, or use a structure-aware extractor.
• Instrument token count, latency, and groundedness before and after enabling compression, and keep a small eval set to catch regressions on every prompt change.
• Rewrite the compression query for multi-turn conversations so it reflects the user's actual current intent, not the raw last message.
• Use a cheap cross-encoder for filtering and reserve LLM calls for extraction, not the other way around.
• Batch chunks into a single compression call where possible instead of one call per chunk.

FAQ

Does contextual compression reduce hallucinations? It reduces one source of hallucination risk, irrelevant context that the model might blend into a wrong answer, but it does not eliminate hallucination on its own. Groundedness still depends on retrieval quality, prompt instructions, and the generator model's own behavior. Treat compression as one layer in a larger accuracy strategy, not a fix by itself.

Is contextual compression the same as summarization? No. Summarization is one possible compression technique, but it rewrites content and carries paraphrase risk. Extraction, the more common default, copies relevant sentences verbatim and does not rewrite anything. When people say "contextual compression" in RAG without qualification, they usually mean extraction or filtering, not summarization.

Does compression slow down my RAG pipeline? It adds a hop, so it adds some latency. Whether the net effect is faster or slower depends on how much it shrinks the generation prompt. A batched compression call over five to ten chunks typically costs less time than the generation savings from a shorter prompt, but you should measure this on your own pipeline rather than assume it.

Can I skip compression and just retrieve smaller chunks instead? Smaller chunks at ingestion time reduce the need for compression but do not eliminate it, because relevance still varies within even a small chunk, and very small chunks lose surrounding context that helps the model interpret them correctly. Compression and chunk sizing address the same problem from different ends of the pipeline; most production systems use both rather than relying on one.

What is the cheapest way to add contextual compression to an existing RAG pipeline? Start with a filtering-only pass using a cross-encoder reranker model, no LLM calls involved. This removes the clearly irrelevant chunks at near-zero added latency. Add LLM-based extraction only if you find, through the groundedness and token-count metrics above, that filtering alone is not shrinking your context enough.

Should I compress before or after reranking? After. Rerank first to cut a large candidate pool down to a small, high-relevance set, then compress that smaller set. Compressing before reranking wastes compression calls on chunks reranking would discard.