LangChain Multi-Vector Retriever: Handling Summaries and Full Docs

Why your retriever keeps missing the right chunk

If you have built even one production RAG pipeline, you have probably hit this wall: a user asks a perfectly reasonable question, your vector search returns "relevant" chunks, and the answer is still wrong. Not because the LLM hallucinated, but because the retriever handed it a fragment of a table, half a paragraph with no context, or a chunk so generic it matched everything and nothing.

The root problem is that standard vector retrieval makes an assumption that quietly breaks at scale: the thing you embed should be the same thing you return to the LLM. For short, self-contained chunks, that assumption holds up fine. For long documents, dense technical reports, tables, or multi-modal content like slides and scanned PDFs, it falls apart. A 500-word chunk embedded as a single vector loses precision. A one-sentence chunk is precise but starves the LLM of context.

LangChain's multi-vector retriever exists to break that assumption on purpose. It lets you embed one representation of your content — a summary, a hypothetical question, a table description — while storing and returning a completely different representation, like the full original document. This decoupling is one of the more underrated patterns in the retrieval-augmented generation toolkit, and it solves a class of bugs that chunk-tuning alone never will.

In this article, we will build a multi-vector retriever from scratch, cover the three main strategies for generating the "index-time" representation, handle summaries and full documents side by side, and talk through the failure modes you will hit in production. By the end you will have working code you can drop into a real pipeline, not just a toy notebook example.

The core idea: decouple what you search from what you return

Every retriever built on a vector store does two jobs at once by default:

1. It converts content into embeddings so similarity search can find it.
2. It returns that same content back to the caller once it's found.

The multi-vector retriever splits these two jobs apart. You store small, information-dense representations in the vector store for search, and you store the actual content you want the LLM to see in a separate document store, linked by an ID. When a query comes in, LangChain searches the vector store, gets back a list of IDs, and then fetches the full content for those IDs from the document store.

Concretely, this means:

• The vector store only ever holds summaries, hypothetical questions, or other "proxy" text.
• The document store (usually an InMemoryStore, Redis, or a database-backed byte store) holds the real payload: full documents, raw tables, or even image bytes.
• A shared key, typically a doc_id, ties every summary vector back to its parent document.

This is exactly what MultiVectorRetriever in langchain.retrievers.multi_vector implements. It is not a new embedding model or a new vector database — it is a coordination layer between two stores you already understand.

Setting up the two stores

Before writing retrieval logic, you need the two pieces of storage the pattern depends on: a vector store for the searchable representations, and a docstore for the original content.

import uuid
from langchain.retrievers.multi_vector import MultiVectorRetriever
from langchain.storage import InMemoryStore
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

# Vector store holds only the small, searchable representations
vectorstore = Chroma(
    collection_name="summaries",
    embedding_function=OpenAIEmbeddings(model="text-embedding-3-small"),
)

# Docstore holds the original, full-size documents
docstore = InMemoryStore()

id_key = "doc_id"

retriever = MultiVectorRetriever(
    vectorstore=vectorstore,
    docstore=docstore,
    id_key=id_key,
)

Notice that at this point the retriever has no data. MultiVectorRetriever is deliberately unopinionated about how you populate the two stores — that logic is yours to write, which is also where most of the design decisions live.

InMemoryStore is fine for prototyping, but in production you will typically swap it for something durable. LangChain's storage module ships adapters for Redis, local file system storage, and SQL-backed key-value stores, all of which implement the same BaseStore interface, so the retriever code above does not change.

Strategy one: summaries as the searchable proxy

The most common use of multi-vector retrieval is summarization. Long documents — think 20-page PDFs, legal contracts, or multi-section technical docs — are hard to embed well as a single vector because the embedding has to compress too much meaning into one point in vector space. A summary captures the gist in far fewer tokens, which produces a tighter, more discriminative embedding.

Here is the full pattern: split documents, summarize each one with an LLM, embed the summaries, and store the originals in the docstore.

from langchain_core.documents import Document
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

summarize_prompt = ChatPromptTemplate.from_template(
    "Summarize the following document in 3-4 sentences, "
    "capturing the key entities, numbers, and conclusions:\n\n{doc}"
)

summarize_chain = (
    {"doc": lambda x: x.page_content}
    | summarize_prompt
    | ChatOpenAI(model="gpt-4o-mini", temperature=0)
    | StrOutputParser()
)

def index_documents(docs: list[Document], retriever: MultiVectorRetriever):
    doc_ids = [str(uuid.uuid4()) for _ in docs]

    summaries = summarize_chain.batch(docs, {"max_concurrency": 5})

    summary_docs = [
        Document(page_content=summary, metadata={id_key: doc_ids[i]})
        for i, summary in enumerate(summaries)
    ]

    # Embed and index only the summaries
    retriever.vectorstore.add_documents(summary_docs)

    # Store the full original documents, keyed by the same IDs
    retriever.docstore.mset(list(zip(doc_ids, docs)))

    return doc_ids

The mset call is the piece people forget. It writes (doc_id, original_document) pairs into the docstore using the exact same IDs you attached to the summary metadata. That shared ID is the only thing linking the two stores together — get it wrong and retrieval will silently return None or throw a key error deep in a chain.

Querying is now trivial, because MultiVectorRetriever handles the two-hop lookup internally:

results = retriever.invoke("What were the Q3 findings on customer churn?")

for doc in results:
    print(doc.page_content[:200])

Under the hood, invoke runs a similarity search against vectorstore, pulls out the doc_id from each hit's metadata, deduplicates those IDs, and calls docstore.mget(ids) to fetch the full documents. The LLM downstream never sees the summary at all — it sees the full original text, which is exactly what you want for grounded, detailed answers.

Strategy two: hypothetical questions

Summaries help with information density, but they still describe the document from the *author's* perspective. Users ask questions, and questions do not always phrase things the way a summary would. A second, often more effective, strategy is to generate synthetic questions that the document could answer, and embed those instead.

from pydantic import BaseModel, Field

class HypotheticalQuestions(BaseModel):
    questions: list[str] = Field(
        description="A list of 3 hypothetical questions this document could answer"
    )

question_prompt = ChatPromptTemplate.from_template(
    "Generate exactly 3 hypothetical questions that the following document "
    "could be used to answer. Focus on questions a user would realistically ask:\n\n{doc}"
)

question_chain = (
    {"doc": lambda x: x.page_content}
    | question_prompt
    | ChatOpenAI(model="gpt-4o-mini", temperature=0).with_structured_output(
        HypotheticalQuestions
    )
)

def index_with_questions(docs: list[Document], retriever: MultiVectorRetriever):
    doc_ids = [str(uuid.uuid4()) for _ in docs]
    hypothetical_results = question_chain.batch(docs, {"max_concurrency": 5})

    question_docs = []
    for i, result in enumerate(hypothetical_results):
        for question in result.questions:
            question_docs.append(
                Document(page_content=question, metadata={id_key: doc_ids[i]})
            )

    retriever.vectorstore.add_documents(question_docs)
    retriever.docstore.mset(list(zip(doc_ids, docs)))
    return doc_ids

This is a many-to-one relationship: each original document can produce several question embeddings, all pointing back to the same doc_id. That is completely fine — MultiVectorRetriever deduplicates IDs after the similarity search, so if two of a document's three hypothetical questions both match the query, you still get that document back exactly once, not twice.

In practice, hypothetical questions tend to outperform summaries when your users ask narrow, specific questions ("what is the maximum retry count for the webhook handler?") rather than broad ones ("tell me about the webhook system"). Many teams index both summaries and hypothetical questions in the same vector store, pointed at the same docstore, and let whichever representation scores higher for a given query win.

Strategy three: smaller chunks pointing to larger parents

A third variant, sometimes called the "parent document retriever" pattern, uses the same underlying machinery but splits differently. Instead of summarizing, you split each document into small child chunks for embedding, while storing the larger parent chunk (or the whole document) in the docstore.

from langchain_text_splitters import RecursiveCharacterTextSplitter

parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=0)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=20)

def index_parent_child(docs: list[Document], retriever: MultiVectorRetriever):
    parent_docs = parent_splitter.split_documents(docs)
    parent_ids = [str(uuid.uuid4()) for _ in parent_docs]

    all_child_docs = []
    for i, parent in enumerate(parent_docs):
        children = child_splitter.split_documents([parent])
        for child in children:
            child.metadata[id_key] = parent_ids[i]
        all_child_docs.extend(children)

    retriever.vectorstore.add_documents(all_child_docs)
    retriever.docstore.mset(list(zip(parent_ids, parent_docs)))
    return parent_ids

This is cheaper than summarization because it skips the LLM call entirely — it's pure text splitting. The trade-off is that small chunks are noisier as search targets than summaries or hypothetical questions, since they are still raw text rather than a distilled representation. Many teams start here because it is free and fast, then move to summaries or hypothetical questions for the subset of documents where retrieval quality actually matters.

If you want this exact behavior without wiring the two stores yourself, LangChain also ships ParentDocumentRetriever, which wraps this child-to-parent splitting logic directly. It is worth knowing both exist: ParentDocumentRetriever is the batteries-included version of this one specific strategy, while MultiVectorRetriever is the general-purpose primitive that also powers summaries, hypothetical questions, and multi-modal indexing.

Handling tables and images the same way

The reason this pattern matters beyond text summarization is that it generalizes cleanly to non-text content. Tables embedded as raw markdown or CSV text tend to produce poor embeddings — the semantic content of a table is in its structure and numbers, not in prose, and embedding models are tuned on natural language. The fix is the same trick: generate a natural-language description of the table, embed that description, and store the original table (or the image of it) in the docstore.

table_summary_prompt = ChatPromptTemplate.from_template(
    "You are analyzing a table extracted from a document. Write a concise "
    "natural-language description of what this table shows, including any "
    "notable trends or figures:\n\n{table}"
)

table_summary_chain = (
    {"table": lambda x: x}
    | table_summary_prompt
    | ChatOpenAI(model="gpt-4o-mini", temperature=0)
    | StrOutputParser()
)

def index_tables(raw_tables: list[str], retriever: MultiVectorRetriever):
    table_ids = [str(uuid.uuid4()) for _ in raw_tables]
    descriptions = table_summary_chain.batch(raw_tables, {"max_concurrency": 5})

    description_docs = [
        Document(page_content=desc, metadata={id_key: table_ids[i]})
        for i, desc in enumerate(descriptions)
    ]

    retriever.vectorstore.add_documents(description_docs)
    # Store the raw table markdown/HTML as the original payload
    retriever.docstore.mset(list(zip(table_ids, raw_tables)))
    return table_ids

The same pattern extends to images: a multimodal model generates a text caption or description of a chart or diagram, that caption gets embedded, and the raw image bytes (or a base64 string, or a file path) sit in the docstore. When the retriever matches the caption, it hands back the original image to a vision-capable model for the final answer — the embedding model never has to "see" the image at all, which sidesteps a whole category of multimodal embedding limitations.

Combining representations in one retriever

Nothing stops you from mixing strategies for the same corpus. A common production setup indexes summaries, hypothetical questions, and raw child chunks all in the same vector store, all pointing back to the same parent documents in the docstore:

def index_multi_strategy(docs: list[Document], retriever: MultiVectorRetriever):
    doc_ids = [str(uuid.uuid4()) for _ in docs]
    retriever.docstore.mset(list(zip(doc_ids, docs)))

    summaries = summarize_chain.batch(docs, {"max_concurrency": 5})
    summary_docs = [
        Document(page_content=s, metadata={id_key: doc_ids[i]})
        for i, s in enumerate(summaries)
    ]

    questions_batch = question_chain.batch(docs, {"max_concurrency": 5})
    question_docs = [
        Document(page_content=q, metadata={id_key: doc_ids[i]})
        for i, result in enumerate(questions_batch)
        for q in result.questions
    ]

    child_docs = []
    for i, doc in enumerate(docs):
        for chunk in child_splitter.split_documents([doc]):
            chunk.metadata[id_key] = doc_ids[i]
            child_docs.append(chunk)

    retriever.vectorstore.add_documents(summary_docs + question_docs + child_docs)
    return doc_ids

This is more expensive to build — you are paying for two LLM calls per document plus a splitting pass — but it widens the surface area for a match. A query that would have missed a dense summary might land squarely on a hypothetical question or a specific child chunk instead. Since MultiVectorRetriever deduplicates by doc_id before returning results, having three representations of the same document in the vector store does not triple the noise in your final output; it just triples your chances of hitting a good match.

Tuning retrieval behavior: search type and result count

MultiVectorRetriever exposes the same knobs you'd expect from any LangChain retriever, because under the hood it delegates the actual similarity search to whatever vector store you passed in.

retriever = MultiVectorRetriever(
    vectorstore=vectorstore,
    docstore=docstore,
    id_key=id_key,
    search_type="mmr",       # use maximal marginal relevance instead of pure similarity
    search_kwargs={"k": 4},  # number of summary/proxy matches to pull before dedup
)

Switching to "mmr" (maximal marginal relevance) is worth trying if you are indexing hypothetical questions, since a plain similarity search can return three near-duplicate questions that all point to the same document, wasting your k budget. MMR explicitly penalizes redundancy among the top results, which pushes the retriever toward pulling proxies from *different* underlying documents rather than three variations of the same one.

Keep in mind that k refers to the number of matches in the proxy search, not the number of documents returned. If you index three hypothetical questions per document and set k=6, you could get anywhere from two to six unique documents back, depending on how much overlap there is in which document each question maps to. This trips people up when they assume k directly controls context length passed to the LLM — measure it, don't assume it.

Common pitfalls and how to avoid them

A few mistakes show up repeatedly when teams adopt this pattern for the first time.

• ID mismatch between stores. If you generate doc_ids for the summary pass and different IDs for the docstore pass, retrieval will fail silently and return None for every "hit." Always generate IDs once, before either store is touched, and reuse the same list.
• Forgetting to persist the docstore. InMemoryStore is wiped on restart. If you index thousands of documents into it and only persist the vector store, you will have summary embeddings pointing to IDs that no longer resolve to anything after a redeploy. Use a RedisStore or a file-system-backed store for anything beyond local testing.
• Summarizing with a model that drops key entities. A summary that says "the report discusses several performance metrics" without naming which ones is nearly useless for retrieval — it is too generic to discriminate between documents. Prompt explicitly for named entities, numbers, and specific nouns in your summarization chain.
• Not deduplicating downstream. If you index multiple representations of one document, always check that you're relying on MultiVectorRetriever's built-in ID deduplication rather than re-deduplicating manually with a different key — mixing both often produces subtly wrong results when metadata is inconsistent across representations.
• Treating this as a free lunch. Every summary and every hypothetical question is an LLM call at index time. For a corpus of 50,000 documents, that is 50,000 (or more, if you generate multiple questions per document) LLM calls before you've answered a single user query. Budget for this, batch aggressively, and consider caching intermediate summaries so re-indexing after a docstore migration does not force you to regenerate everything from scratch.

Evaluating whether the extra indexing cost is worth it

Because every strategy beyond raw child chunks adds LLM calls at index time, it is worth measuring whether the added retrieval quality actually justifies the cost before you commit a whole pipeline to it. A simple way to do this without building a full evaluation harness is to keep a small, hand-labeled set of query-to-document pairs pulled from real support tickets or real user questions, and run each indexing strategy against the same corpus.

eval_pairs = [
    {"query": "what is the refund window for annual plans?", "expected_doc_id": "doc_12"},
    {"query": "how do I rotate the API key for a workspace?", "expected_doc_id": "doc_47"},
    # ... more labeled pairs
]

def hit_rate(retriever: MultiVectorRetriever, pairs: list[dict], k: int = 4) -> float:
    hits = 0
    for pair in pairs:
        results = retriever.invoke(pair["query"])
        returned_ids = [r.metadata.get("source_id") for r in results[:k]]
        if pair["expected_doc_id"] in returned_ids:
            hits += 1
    return hits / len(pairs)

Running this against a summary-only index, a hypothetical-question index, and a combined index on the same 30-50 labeled pairs will usually tell you within an hour whether the added complexity is worth shipping. In our own testing across different corpora, hypothetical questions tend to win on narrow factual queries, summaries tend to win on broad "explain this concept" queries, and the combined index is rarely worse than either alone — it is just more expensive to build and maintain. Treat that expense as a real line item, not an afterthought, especially if your corpus grows continuously and you need to re-index on a schedule.

It's also worth tracking this metric over time rather than treating it as a one-time check. Document corpora drift — new content gets added, old content gets deprecated, and the kinds of questions users ask shift as your product evolves. A hit rate that looked great at launch can quietly degrade six months later if nobody is watching it. Wiring this evaluation into a lightweight CI check that runs whenever the indexing pipeline changes catches regressions before they reach users, rather than after a support ticket flags a wrong answer.

Choosing a docstore backend for production

The examples above all use InMemoryStore because it requires no setup, but it is almost never the right choice once you are past local prototyping. The two properties you actually care about are durability and lookup latency, and different backends make different trade-offs on both.

• Local file system store. LocalFileStore writes each value to disk as a separate file keyed by ID. It survives restarts and requires no external service, which makes it a reasonable middle ground for small deployments or single-machine batch jobs. It does not scale well past a few hundred thousand keys because directory listings and file-open overhead start to dominate.
• Redis-backed store. RedisStore gives you sub-millisecond lookups and horizontal scaling, at the cost of running (and paying for) a Redis instance. This is the most common production choice because docstore access is a hot path — every single retrieval call triggers an mget against it, so latency here directly affects your end-to-end response time.
• SQL-backed store. If you already have a relational database in your stack, a SQL-backed BaseStore implementation lets you reuse existing infrastructure and get transactional guarantees around writes, which matters if your indexing pipeline needs to be resumable after a partial failure.

Whichever backend you choose, remember that the docstore and the vector store can fall out of sync if either write fails independently. In the indexing functions above, add_documents and mset are two separate calls with no shared transaction. If the process crashes between them, you can end up with summary vectors that point to IDs the docstore has never heard of, or documents in the docstore that no vector ever points to. For any indexing job that matters, wrap both calls in a retry-safe function and log the doc_ids batch before you start, so a crashed job can be resumed or cleanly rolled back rather than left in an inconsistent half-written state.

A note on chunk size for the underlying documents

Multi-vector retrieval solves the *search* half of the precision-versus-context trade-off, but it does not eliminate the need to think about chunk size for the documents you store in the docstore. If your "full document" is genuinely a 40-page PDF, handing the entire thing to the LLM on every retrieval hit will blow through context windows fast and bury the actually-relevant passage in noise the model has to read past.

A pragmatic middle ground is to split source documents into medium-sized sections — a few hundred to a couple thousand tokens each — before summarizing, so each docstore entry is a coherent section rather than an entire book. You get the precision benefit of a targeted summary for search, and the returned content is still small enough that the LLM can actually use all of it rather than needing yet another summarization pass downstream. This is exactly what the parent-child chunk strategy does implicitly, and it's worth borrowing that sizing discipline even when you're indexing summaries or hypothetical questions instead of raw child chunks.

Wrapping up

The multi-vector retriever is one of those patterns that looks like a minor implementation detail until you actually need it, at which point it becomes the difference between a RAG system that works on toy examples and one that survives contact with real documents. By separating what gets embedded from what gets returned, you can shrink your searchable representations down to exactly the information density that similarity search needs, while still handing the LLM the full, ungutted context it needs to answer well.

Start simple: pick one strategy, probably summaries, get the two-store wiring correct, and verify retrieval quality on a handful of real queries before you add hypothetical questions or table/image handling on top. The ID-matching discipline is the only genuinely tricky part — once that's solid, layering additional representations is mostly a matter of writing another indexing function and pointing it at the same docstore.

If you want to go deeper into this pattern alongside the rest of LangChain's retrieval and chain-composition toolkit — including LCEL, agents, and production deployment patterns — our LangChain Tutorial 2026 course on teachyou.ai walks through multi-vector retrieval, parent-document retrievers, and multi-modal RAG pipelines with hands-on projects built from the ground up.