The Parent Document Retriever Pattern in RAG
A parent document retriever is a retrieval pattern that splits documents into small chunks for embedding and search, but returns the larger parent chunk (or the full original document) to the language model at generation time. It solves a tension that every RAG builder eventually hits: small chunks give you precise semantic matches, but small chunks alone often don't carry enough context for the model to answer well. The parent document retriever decouples what you search from what you feed to the model.
This pattern goes by a few names depending on the framework: LangChain calls it ParentDocumentRetriever, LlamaIndex implements the same idea through its AutoMergingRetriever and sentence-window node parsers, and plenty of teams build it by hand with two tables in Postgres. The mechanics are the same everywhere: index fine, retrieve coarse.
Why small chunks alone fail in RAG
Embedding models compress text into a fixed-size vector, and that compression has a sweet spot. A chunk of 150-300 tokens tends to produce an embedding that's tightly aligned with a specific topic or claim. Chunk at 2000 tokens and the embedding becomes an average of several different ideas, so a query about one narrow detail buried in that chunk won't score as high a similarity as it should. This is why most RAG tutorials tell you to keep chunks small.
But small chunks create a second problem at generation time. A 200-token chunk pulled from the middle of a document is often missing the antecedent for a pronoun, the definition of a term introduced two paragraphs earlier, or the table header that gives the surrounding numbers meaning. The model gets a fragment and either hallucinates the missing context or gives a vague, hedged answer.
You can see this trade-off directly:
- Small chunks (100-300 tokens): high retrieval precision, low generation quality when used alone
- Large chunks (1500+ tokens): low retrieval precision (diluted embeddings), high generation quality when matched
- Parent document retriever: high retrieval precision AND high generation quality, at the cost of extra storage and lookup complexity
The parent document retriever pattern exists because "small chunks for search, large chunks for reading" isn't actually a contradiction if you store both and only couple them at query time.
How the parent document retriever works
The architecture has three moving parts.
1. A document store keyed by parent ID. This holds the full parent documents (or large parent chunks, say 1500-2000 tokens each), typically in a plain key-value store, a Postgres table, or even local disk. This store is not searched semantically. It's a lookup table.
2. A vector store holding child chunks. Each parent is split into smaller child chunks (150-400 tokens), each child chunk is embedded, and each embedding is tagged with metadata pointing back to its parent's ID. This is the only thing your similarity search touches.
3. A retriever that joins the two. At query time: embed the user's query, run similarity search against the child vector store, collect the parent IDs from the top matches, deduplicate them, and fetch the full parent documents from the document store. Those parents (not the child chunks) get passed into the prompt.
The retrieval flow looks like this:
User query
|
v
Embed query -> similarity search over child chunk vectors
|
v
Top-k child chunks matched (each tagged with parent_id)
|
v
Deduplicate parent_ids
|
v
Fetch full parent documents from docstore
|
v
Parent documents inserted into LLM contextNote what never happens: the child chunk text itself is never sent to the LLM. It exists purely as a high-precision search index that points at something bigger.
Two variants: fixed parents vs. hierarchical parents
There are two common ways to define "parent."
Fixed-size parent chunking. You split the original document into large parent chunks of a set size (say 2000 tokens with some overlap), then split each parent into small child chunks. This is what LangChain's ParentDocumentRetriever does out of the box when you pass both a parent_splitter and a child_splitter.
Whole-document parents. Instead of splitting into parent chunks at all, the "parent" is the entire source document, and only the child chunks are split out for embedding. This works well when your source documents are already reasonably sized, like individual API reference pages, individual support tickets, or individual product spec sheets. It's the simplest version of the pattern and often the one worth starting with.
Hierarchical (multi-level) parents. Some implementations go a level further: sentence -> paragraph -> section -> document, with each level acting as the "parent" of the level below it. LlamaIndex's auto-merging retriever formalizes this: if enough child nodes under the same parent are retrieved, it merges them and returns the parent node instead of the individual children. This gives you a tunable "how much context is enough" threshold rather than an all-or-nothing swap.
For most production RAG systems, fixed-size or whole-document parents get you 90% of the benefit with a fraction of the complexity. Reach for the hierarchical version only once you've measured that a two-level split isn't enough.
Building it with LangChain
Here's a working implementation using LangChain's built-in ParentDocumentRetriever, an in-memory docstore for parents, and Chroma for the child vector index.
from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain_community.document_loaders import DirectoryLoader
# Load raw documents
loader = DirectoryLoader("./docs", glob="**/*.md")
docs = loader.load()
# Parent chunks: large, for context
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
# Child chunks: small, for embedding and search
child_splitter = RecursiveCharacterTextSplitter(chunk_size=300, chunk_overlap=50)
# Vector store for child embeddings only
vectorstore = Chroma(
collection_name="child_chunks",
embedding_function=OpenAIEmbeddings(model="text-embedding-3-small"),
persist_directory="./chroma_child",
)
# Key-value store for full parent documents
docstore = InMemoryStore()
retriever = ParentDocumentRetriever(
vectorstore=vectorstore,
docstore=docstore,
child_splitter=child_splitter,
parent_splitter=parent_splitter,
)
retriever.add_documents(docs)
results = retriever.invoke("How does rate limiting work on the webhook endpoint?")
for r in results:
print(len(r.page_content), r.metadata.get("source"))InMemoryStore is fine for prototyping but won't survive a process restart. For anything you deploy, swap it for LangChain's RedisStore, or write a thin custom docstore backed by Postgres. The BaseStore interface only needs mget, mset, mdelete, and yield_keys, so wrapping a SQL table takes about twenty lines.
from langchain.storage.encoder_backed import EncoderBackedStore
from langchain_core.stores import BaseStore
import psycopg2
import pickle
class PostgresDocStore(BaseStore):
def __init__(self, dsn: str):
self.conn = psycopg2.connect(dsn)
def mget(self, keys):
with self.conn.cursor() as cur:
cur.execute(
"SELECT key, value FROM parent_docs WHERE key = ANY(%s)", (keys,)
)
rows = dict(cur.fetchall())
return [pickle.loads(rows[k]) if k in rows else None for k in keys]
def mset(self, key_value_pairs):
with self.conn.cursor() as cur:
for key, value in key_value_pairs:
cur.execute(
"INSERT INTO parent_docs (key, value) VALUES (%s, %s) "
"ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value",
(key, pickle.dumps(value)),
)
self.conn.commit()
def mdelete(self, keys):
with self.conn.cursor() as cur:
cur.execute("DELETE FROM parent_docs WHERE key = ANY(%s)", (keys,))
self.conn.commit()
def yield_keys(self, prefix=None):
with self.conn.cursor() as cur:
cur.execute("SELECT key FROM parent_docs")
for (key,) in cur.fetchall():
yield keySwap docstore=InMemoryStore() for docstore=PostgresDocStore(dsn) and the rest of the pipeline is unchanged.
Building it by hand, without a framework
If you're not using LangChain, the pattern is simple enough to hand-roll with any vector database. Here's the shape using a raw Postgres + pgvector setup.
CREATE TABLE parents (
id UUID PRIMARY KEY,
content TEXT NOT NULL,
source TEXT
);
CREATE TABLE child_chunks (
id UUID PRIMARY KEY,
parent_id UUID REFERENCES parents(id),
content TEXT NOT NULL,
embedding VECTOR(1536)
);
CREATE INDEX ON child_chunks USING hnsw (embedding vector_cosine_ops);Indexing:
import uuid
def index_document(text: str, source: str, embed_fn, conn):
parent_id = uuid.uuid4()
conn.execute(
"INSERT INTO parents (id, content, source) VALUES (%s, %s, %s)",
(parent_id, text, source),
)
child_chunks = split_into_chunks(text, chunk_size=300, overlap=50)
for chunk in child_chunks:
embedding = embed_fn(chunk)
conn.execute(
"INSERT INTO child_chunks (id, parent_id, content, embedding) "
"VALUES (%s, %s, %s, %s)",
(uuid.uuid4(), parent_id, chunk, embedding),
)
conn.commit()Retrieval:
def parent_document_retrieve(query: str, embed_fn, conn, k=5):
query_embedding = embed_fn(query)
rows = conn.execute(
"""
SELECT DISTINCT ON (parent_id) parent_id
FROM child_chunks
ORDER BY parent_id, embedding <=> %s
LIMIT %s
""",
(query_embedding, k),
).fetchall()
parent_ids = [r[0] for r in rows]
if not parent_ids:
return []
parents = conn.execute(
"SELECT id, content, source FROM parents WHERE id = ANY(%s)",
(parent_ids,),
).fetchall()
return parentsTwo details matter here. First, ranking is done on child chunk similarity, then deduplicated to unique parent IDs, so the ordering of parents should follow the best-scoring child under each parent, not an arbitrary order. Adjust the SQL to ORDER BY parent_id, embedding <=> query_embedding followed by a second pass that sorts the deduplicated parents by their best child's distance if you need strict ranking. Second, k here means "top-k child chunks," not "top-k parents." If several of your top matches share a parent, you'll end up with fewer unique parents than k, which is usually fine, but size your k a bit larger than the number of parents you actually want returned.
Sizing chunks: parent and child
There's no universal number, but these starting points work for most technical and business documentation:
- Child chunks: 150-400 tokens, 0-15% overlap. Small enough that each one is topically coherent, large enough that it isn't just a sentence fragment losing meaning.
- Parent chunks: 1000-2500 tokens, or a whole document if documents are naturally under ~3000 tokens. Large enough to answer follow-up questions about the surrounding context without a second retrieval round.
- Ratio: aim for 4-8 child chunks per parent. Fewer than that and the parent split barely helps recall; more than that and you're indexing a lot of near-duplicate embeddings per parent.
The one thing worth measuring, not guessing: run retrieval eval (recall@k against a labeled query set) at two or three child sizes before committing. A 200-token child chunk and a 400-token child chunk can produce meaningfully different recall depending on how dense your source documents are.
When to use it and when not to
Use a parent document retriever when your source documents are long relative to a typical answer, when users ask questions that need surrounding context (a step in a procedure, a clause in a contract, a function in the middle of a class), and when your corpus doesn't fit entirely inside a single context window anyway.
Skip it when your documents are already short (FAQ entries, short support macros, single-paragraph glossary terms) since chunking them at all adds no value. Also skip it, or use a small parent size, if your embedding budget for context is genuinely tight; sending 5 parents at 2000 tokens each is 10,000 tokens of context before the model has written a word, which matters if you're running a low-cost model or a tight latency budget.
It also composes well with other RAG techniques rather than replacing them. You can layer reranking on top: retrieve child chunks with the vector search, rerank the child chunks with a cross-encoder, then expand only the top reranked results to their parents. This keeps the expensive rerank step cheap (small chunks) while still delivering large context to the model.
Common mistakes
- Embedding the parent chunks too. Some implementations accidentally index both parent and child text into the same vector store. This dilutes search quality and doubles storage for no benefit. Only child chunks should ever be embedded.
- No deduplication on the parent join. If you skip the
DISTINCT ON(or equivalent), a query that matches four child chunks under the same parent will return that parent four times, wasting context tokens on a repeated document. - Parent chunks too large for your context budget. If you're retrieving 8-10 parents at 2000+ tokens, do the multiplication before you deploy. It's an easy way to blow past your model's effective context or your latency SLA.
- Forgetting to persist the docstore.
InMemoryStore(or any in-process dictionary) disappears on restart. If your parent lookups return empty after a redeploy, this is almost always why. - Static parent-child mapping when documents update. If a source document changes, you need to re-chunk and re-embed both levels and delete the old child chunks tied to the old parent ID, or you'll retrieve stale children pointing at an outdated (or missing) parent.
FAQ
Is parent document retriever the same as hierarchical chunking? They overlap but aren't identical. Parent document retrieval is specifically about decoupling the unit you search from the unit you return, typically two levels (child, parent). Hierarchical chunking usually refers to building three or more levels (sentence, paragraph, section, document) and can be used as the basis for a parent document retriever, but the core pattern only requires two levels to work.
Does this increase my vector database costs? It increases the number of vectors you store, since you're embedding smaller chunks and therefore more of them per document, but it doesn't increase the size of what you store in the document store (parents are typically stored as plain text or JSON, not vectors, so they're cheap). The main cost increase is more embedding calls during indexing and a slightly larger vector index.
Can I use this with LlamaIndex instead of LangChain? Yes. LlamaIndex's equivalent is the auto-merging retriever combined with a HierarchicalNodeParser, which builds the same parent-child relationship and merges children back into their parent node when enough of them are retrieved together. The concept is identical even though the API shape differs.
How do I choose k for the child chunk search? Set it higher than the number of unique parents you actually want to see, since multiple top matches often share a parent. If you want 5 distinct parent documents back, searching for the top 15-20 child chunks and then deduplicating down to parents is a reasonable starting ratio, though this depends heavily on how many children each parent has.
Does parent document retrieval replace reranking? No, they solve different problems. Retrieval precision (finding the right chunks) and context sufficiency (having enough surrounding text to answer well) are separate concerns. Many production systems use both: rerank at the child level to sharpen precision, then expand the reranked results to their parents before generation.
What happens if two child chunks from different parents both match well? Both parents get retrieved and both go into the context, ranked by their best-matching child chunk's similarity score. This is normal and often desirable when an answer genuinely spans two source documents.
BootcampA 30-day guided bootcamp: build, harden and ship a production autonomous agent from scratch.
AI AgentsUnderstand how AI agents really work: the loop, the tools, the memory, and why most agent projects fail.