LangChain Contextual Compression: Trimming Retrieved Documents

Your retriever is returning garbage, and you don't even know it

You built a RAG pipeline. You chunked your documents, embedded them, stored them in a vector database, and wired up a retriever. It works — sort of. The answers are okay, but not great. Sometimes the model rambles, sometimes it misses the point buried in paragraph four of a five-paragraph chunk, and your token bill keeps creeping up every time you widen k to "just retrieve a bit more context."

Here's the uncomfortable truth: standard similarity search retrieves whole chunks, not relevant sentences. A chunk might score high on cosine similarity because one sentence out of ten matches the query, while the other nine sentences are noise — irrelevant boilerplate, unrelated tangents, or repeated headers. You pass all ten sentences to the LLM anyway, because that's how the chunk was stored. The model has to wade through the noise to find the signal, and every token of noise costs you money and attention budget.

This is exactly the problem contextual compression in LangChain was built to solve. Instead of accepting documents as retrieved, you post-process them — filtering out irrelevant documents entirely, extracting only the relevant spans from the ones you keep, or both. The result is a retriever that behaves like a much smarter one, without touching your embeddings or vector store.

In this article, we'll walk through how LangChain's ContextualCompressionRetriever works, build it with real code using LLMChainExtractor, LLMChainFilter, and embeddings-based filters, chain multiple compressors into a pipeline, and talk about when compression helps and when it just adds latency for no gain.

What contextual compression actually means

"Compression" here doesn't mean gzip. It means reducing retrieved documents down to the parts that are actually relevant to the query, using an LLM or a lightweight model as the judge. LangChain implements this as a wrapper pattern: you take a base retriever (any retriever — FAISS, Chroma, Pinecone, a BM25 retriever, whatever) and wrap it in a ContextualCompressionRetriever. That wrapper intercepts the documents coming back from the base retriever and runs them through a BaseDocumentCompressor before returning them to the caller.

There are three broad strategies for the compressor:

• Extraction — keep the document, but strip it down to only the sentences relevant to the query. This is what LLMChainExtractor does.
• Filtering — keep the whole document or discard it entirely, based on a relevance judgment. No text is trimmed within a kept document. This is LLMChainFilter and EmbeddingsFilter.
• Pipeline compression — chain several compressors and transformers together (for example, split into smaller pieces, deduplicate with embeddings, then filter by relevance) using DocumentCompressorPipeline.

The key architectural point: compression happens after retrieval, not instead of it. Your vector search still runs first and does the heavy lifting of narrowing millions of documents down to a handful of candidates. Compression is a second pass that cleans up what similarity search handed you.

Setting up a baseline retriever

Before adding compression, you need a base retriever to wrap. Let's set up a simple FAISS-backed retriever over some sample documents so the compression examples have something real to operate on.

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import FAISS
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Load and split a source document into chunks
loader = TextLoader("company_handbook.txt")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=100,
)
chunks = splitter.split_documents(documents)

# Embed and store the chunks
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = FAISS.from_documents(chunks, embeddings)

base_retriever = vectorstore.as_retriever(
    search_kwargs={"k": 6}
)

# Baseline query without any compression
query = "What is the policy on remote work reimbursement?"
docs = base_retriever.invoke(query)

for i, doc in enumerate(docs):
    print(f"--- Document {i+1} ({len(doc.page_content)} chars) ---")
    print(doc.page_content[:200])

Run this against a real handbook and you'll typically notice the same pattern: two or three chunks are genuinely about remote work reimbursement, and the rest are adjacent topics — travel policy, expense report deadlines, generic HR boilerplate — that got pulled in because k=6 casts a wide net and cosine similarity is not the same as relevance. This is the baseline we'll improve.

Extracting only the relevant parts with LLMChainExtractor

LLMChainExtractor sends each retrieved document to an LLM along with the original query, and asks the LLM to return only the parts of the document that are relevant to answering the query. If nothing in a document is relevant, it returns an empty string, and that document is dropped entirely.

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

compressor = LLMChainExtractor.from_llm(llm)

compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=base_retriever,
)

compressed_docs = compression_retriever.invoke(query)

for i, doc in enumerate(compressed_docs):
    print(f"--- Compressed Document {i+1} ({len(doc.page_content)} chars) ---")
    print(doc.page_content)
    print()

The difference is usually dramatic. Where the base retriever handed back six full chunks totaling several thousand characters, the compressed output might return three trimmed snippets totaling a few hundred characters — each one a direct, on-topic excerpt about reimbursement caps, submission deadlines, and eligible expense categories. Everything else got filtered out at the document level (irrelevant chunks removed) or trimmed at the sentence level (relevant chunks kept but shortened).

Under the hood, LLMChainExtractor uses a prompt template that instructs the LLM to extract verbatim text — it's told not to paraphrase or summarize, only to copy relevant spans. That matters for RAG: you want the exact wording from your source documents, not the LLM's lossy restatement of it, because the final answer-generation step will cite or quote this text.

You can inspect and override that prompt if the default doesn't match your domain:

from langchain.retrievers.document_compressors.chain_extract import (
    prompt_template,
)

print(prompt_template)

If your documents are technical (API references, legal clauses, medical notes), you may want to customize this prompt so the extractor knows to preserve exact numbers, section identifiers, or defined terms rather than trimming them as "boilerplate."

Filtering whole documents with LLMChainFilter

Sometimes you don't want the LLM rewriting or trimming document text at all — you just want a cheap relevance gate that says "keep this chunk" or "drop this chunk," preserving the original text exactly. That's what LLMChainFilter does. It's faster and cheaper than LLMChainExtractor because the LLM only needs to output a yes/no-style decision, not regenerate document text.

from langchain.retrievers.document_compressors import LLMChainFilter

filter_compressor = LLMChainFilter.from_llm(llm)

filter_retriever = ContextualCompressionRetriever(
    base_compressor=filter_compressor,
    base_retriever=base_retriever,
)

filtered_docs = filter_retriever.invoke(query)

print(f"Base retriever returned: {len(docs)} documents")
print(f"Filter retriever returned: {len(filtered_docs)} documents")

for doc in filtered_docs:
    print(doc.page_content[:150], "...\n")

Use LLMChainFilter when you care about token savings from removing irrelevant chunks entirely, but you still want the surviving chunks intact — for example, when downstream logic expects full paragraphs for citation purposes, or when your extraction step already trims aggressively upstream and you just need a coarse filter.

Skipping the LLM call: EmbeddingsFilter

Both extractor and filter approaches above call an LLM for every retrieved document, on every query. That's an LLM call multiplied by k documents, multiplied by every query your application serves — it adds real latency and real cost. If you're running this at any meaningful volume, that overhead matters.

EmbeddingsFilter gives you a middle ground: instead of asking an LLM to judge relevance, it computes the embedding similarity between the query and each document, and drops documents below a similarity threshold. No LLM call, no token cost for the compression step itself — just a vector comparison.

from langchain.retrievers.document_compressors import EmbeddingsFilter

embeddings_filter = EmbeddingsFilter(
    embeddings=embeddings,
    similarity_threshold=0.76,
)

embeddings_retriever = ContextualCompressionRetriever(
    base_compressor=embeddings_filter,
    base_retriever=base_retriever,
)

result = embeddings_retriever.invoke(query)
print(f"Documents passing threshold: {len(result)}")

The tradeoff is precision: embedding similarity is a blunter instrument than an LLM reading the query and the document together. It can't reason about negation ("policy does NOT cover reimbursement for personal vehicle mileage"), and it can't do the kind of nuanced relevance judgment an LLM can. But it's fast, cheap, and works well as either a standalone filter for high-volume applications or as a first-pass filter in a pipeline before a more expensive extractor runs on the survivors.

Building a real pipeline: DocumentCompressorPipeline

The most useful pattern in practice isn't picking one compressor — it's chaining several together, each doing a cheap job before the next, more expensive step runs. LangChain's DocumentCompressorPipeline lets you compose document transformers (text splitters, redundancy filters) and compressors (relevance filters, extractors) into a single sequential pipeline.

A common, effective ordering:

1. Split large chunks into smaller pieces (so extraction and filtering operate on finer-grained text).
2. Deduplicate near-identical pieces using embeddings (retrieval often surfaces overlapping chunks).
3. Filter by embedding similarity (cheap, removes obviously irrelevant pieces).
4. Extract relevant spans with an LLM (expensive, but now only running on a much smaller, pre-filtered set).

from langchain.retrievers.document_compressors import (
    DocumentCompressorPipeline,
    EmbeddingsFilter,
)
from langchain_community.document_transformers import EmbeddingsRedundantFilter
from langchain_text_splitters import CharacterTextSplitter

splitter = CharacterTextSplitter(chunk_size=300, chunk_overlap=0, separator=". ")

redundant_filter = EmbeddingsRedundantFilter(embeddings=embeddings)

relevance_filter = EmbeddingsFilter(
    embeddings=embeddings,
    similarity_threshold=0.70,
)

extractor = LLMChainExtractor.from_llm(llm)

pipeline_compressor = DocumentCompressorPipeline(
    transformers=[splitter, redundant_filter, relevance_filter, extractor]
)

pipeline_retriever = ContextualCompressionRetriever(
    base_compressor=pipeline_compressor,
    base_retriever=base_retriever,
)

final_docs = pipeline_retriever.invoke(query)

for doc in final_docs:
    print(doc.page_content)
    print("---")

Notice the ordering matters for cost. If you ran the LLM extractor first and the embeddings filter last, you'd pay the full LLM cost on every document before discarding most of them — wasted spend. By pushing the free and cheap steps (splitting, redundancy removal, embedding similarity) to the front, the expensive LLM step only ever touches the small, already-filtered remainder. This is the same principle as putting a cheap WHERE clause before an expensive JOIN in a SQL query planner — filter early, compute late.

Plugging the compression retriever into a RAG chain

Once you have a ContextualCompressionRetriever, it's a drop-in replacement for any other retriever — that's the whole point of LangChain's retriever interface. Wiring it into a retrieval-augmented generation chain requires no special handling.

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

prompt = ChatPromptTemplate.from_template(
    """Answer the question using only the context below.
If the answer isn't in the context, say you don't know.

Context:
{context}

Question: {question}
"""
)

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

rag_chain = (
    {
        "context": compression_retriever | format_docs,
        "question": RunnablePassthrough(),
    }
    | prompt
    | llm
    | StrOutputParser()
)

answer = rag_chain.invoke(
    "What is the policy on remote work reimbursement?"
)
print(answer)

Because the compression retriever returns the same Document objects any other retriever returns, nothing downstream needs to change. You get smaller, cleaner context windows feeding your prompt, which typically means faster generation, lower per-call token cost, and — often the more important benefit — an LLM that isn't distracted by irrelevant text sitting next to the answer.

When compression helps and when it doesn't

Contextual compression is not a free win in every RAG setup. It's worth being honest about the tradeoffs before you drop it into a production pipeline.

Compression tends to help when:

• Your chunks are large (500+ tokens) and topically mixed, so a lot of retrieved text per chunk is irrelevant to any single query.
• You retrieve a high k (6, 10, 20) to improve recall, and need to control the resulting context size before it reaches the LLM.
• Token cost genuinely matters — you're serving high query volume, or using a model priced per token where trimming 70% of retrieved text has a real dollar impact.
• Precision matters more than raw recall — for example, compliance or legal use cases where irrelevant surrounding text could mislead the model into an incorrect citation.

Compression tends to hurt or simply add cost when:

• Your chunks are already small and tightly scoped (for example, one FAQ entry per chunk). There's little noise left to trim, so the extra LLM call is pure overhead.
• Latency is the binding constraint. LLMChainExtractor and LLMChainFilter both add a synchronous LLM round-trip per document before your main generation call even starts. In a chat UI where users expect sub-second responses, that added latency can be worse than the quality gain.
• You need the exact original chunk text preserved for auditing or exact citation back to source — LLMChainExtractor rewrites document boundaries, which can break strict provenance requirements unless you also track original chunk IDs in metadata.

A pragmatic default: start with EmbeddingsFilter alone. It's nearly free, it removes the worst offenders, and it requires no additional LLM budget. Only escalate to LLMChainFilter or LLMChainExtractor — or a full pipeline — once you've measured that embedding-based filtering isn't cutting it for your specific queries.

It also helps to think about compression as sitting on a spectrum with reranking, since teams often confuse the two. A reranker (like a cross-encoder model) reorders the same set of documents by relevance but doesn't touch their content or count — you still send all k documents downstream, just in a better order. Compression, by contrast, actually reduces what gets sent — fewer documents, shorter documents, or both. In a mature RAG pipeline you often want both: retrieve a wide candidate set, rerank to push the best matches to the top, then compress to trim what actually reaches the prompt. Skipping straight to compression without addressing ranking quality first can mean you're carefully trimming documents that shouldn't have been retrieved so prominently in the first place.

It's also worth deciding early whether compression runs synchronously in the request path or as a batch/offline step. For a chat application answering one query at a time, compression has to run inline — the user is waiting, so every LLM call in the compression stage adds to time-to-first-token. For a system that pre-computes context for a known, bounded set of queries (an FAQ bot with anticipated questions, or a nightly re-indexing job), you can run compression ahead of time and cache the compressed output, which removes the latency concern almost entirely. Recognizing which situation you're in changes which compressor is affordable.

Measuring whether compression is actually working

Don't take it on faith that compression improves your RAG quality — measure it. A simple before/after harness compares token counts and, ideally, answer quality on a fixed evaluation set.

import tiktoken

encoding = tiktoken.encoding_for_model("gpt-4o-mini")

def count_tokens(docs):
    return sum(len(encoding.encode(doc.page_content)) for doc in docs)

test_queries = [
    "What is the policy on remote work reimbursement?",
    "How many vacation days do new hires get?",
    "What is the process for expense report approval?",
]

for q in test_queries:
    base = base_retriever.invoke(q)
    compressed = compression_retriever.invoke(q)

    base_tokens = count_tokens(base)
    compressed_tokens = count_tokens(compressed)
    reduction = 100 * (1 - compressed_tokens / base_tokens) if base_tokens else 0

    print(f"Query: {q}")
    print(f"  Base tokens: {base_tokens}, Compressed tokens: {compressed_tokens}")
    print(f"  Reduction: {reduction:.1f}%")
    print(f"  Base docs: {len(base)}, Compressed docs: {len(compressed)}")
    print()

Run this across a representative sample of your real queries, not just one hand-picked example. Token reduction alone doesn't prove quality improved — pair it with a small held-out set of question/expected-answer pairs and check that the compressed retriever still surfaces the text needed to answer correctly. If you find compression is stripping out passages your evaluation set actually needs, that's a signal your extraction prompt is too aggressive, or your similarity threshold in EmbeddingsFilter is set too high.

Common pitfalls to watch for

A few issues come up repeatedly when teams adopt contextual compression:

• Empty results after filtering. If your similarity_threshold in EmbeddingsFilter is set too high, or your LLM filter prompt is too strict, you can end up with zero documents returned for valid queries. Always test edge cases where the answer is only weakly present in the retrieved set.
• Compounding latency in pipelines. Each stage in a DocumentCompressorPipeline that calls an LLM adds latency. If you chain LLMChainFilter and LLMChainExtractor together, you're paying for two LLM passes over your documents before generation even starts. Profile actual wall-clock time under realistic load, not just in a notebook.
• Losing metadata. Some compressors return new Document objects — verify that metadata (source, page number, chunk ID) survives the transformation, since you'll likely want it for citations in the final answer.
• Over-trimming context that the LLM needs for reasoning. Aggressive extraction can strip surrounding context that isn't a direct match for the query but that the LLM needs to interpret the matched sentence correctly (definitions, prior clauses, units). If answers start getting confidently wrong rather than vague, check whether your extractor is cutting too close to the bone.
• Treating compression as a substitute for better chunking. If your base retrieval quality is poor because your chunking strategy is wrong (too large, split mid-sentence, no overlap), compression is a bandage, not a cure. Fix chunking first; compression is for cleaning up chunks that are structurally fine but topically mixed.
• Forgetting to cache repeated queries. If the same or similar questions recur often — support bots and internal knowledge tools both see this constantly — recomputing compression on every single request wastes LLM calls on work you already did. A simple cache keyed on the normalized query (or the retrieved document IDs) can skip the compression step entirely for repeat traffic.
• Not handling the empty-context case in the prompt. If every document gets filtered out, your prompt template still needs to behave sensibly with an empty context variable — otherwise you get a confidently wrong answer instead of an honest "I don't have that information." Test this path explicitly; it's easy to only test the happy path where compression always leaves something behind.

Wrapping up

Contextual compression solves a specific, common problem in RAG systems: retrieved documents carrying more noise than signal. ContextualCompressionRetriever wraps any existing retriever without requiring you to rebuild your vector store, and you have several compressor strategies to choose from depending on your cost and latency budget — EmbeddingsFilter for a cheap first pass, LLMChainFilter for cleaner document-level relevance gating, LLMChainExtractor for precise sentence-level trimming, and DocumentCompressorPipeline when you want to combine several of these in the right order.

The right approach depends on your traffic volume, latency tolerance, and how noisy your chunks actually are — there's no universal setting, which is why measuring token reduction and answer quality on your own queries matters more than copying a default configuration from a tutorial.

If you want to go deeper into building production RAG pipelines — chunking strategies, retrievers, compression, reranking, and evaluation — this is exactly the kind of hands-on work we cover in the LangChain Tutorial 2026 course at TeachYou.ai, with real datasets and real failure cases instead of toy examples.