Tuning RAG Chunk Size and Overlap

Picking the right rag chunk size is one of the highest-leverage decisions you make when building a retrieval pipeline, and most teams set it once and never revisit it. Chunk size determines how much context each embedding represents, how many chunks compete for the top-k slots your language model actually sees, and how much irrelevant text gets dragged into the prompt alongside the answer. Get it wrong and no amount of prompt engineering or model upgrades will fix the retrieval quality ceiling you built for yourself. This article walks through how chunk size and overlap actually affect retrieval, gives you working code to test different strategies, and shows how to measure the tradeoffs instead of guessing.

Why rag chunk size matters more than people think

A chunk is the unit you embed and retrieve. Every design choice downstream, embedding model, vector database, reranker, prompt template, inherits whatever boundaries you drew at chunking time. If a chunk is too large, the embedding becomes a blurry average of several topics, and a query that matches one sentence buried inside it still has to compete on similarity with chunks that are more topically focused. If a chunk is too small, you lose the surrounding context a model needs to answer correctly, and you multiply the number of vectors you have to store, index, and search.

There are three concrete failure modes tied directly to chunk size:

• Semantic dilution: a 2000-token chunk covering three subtopics produces an embedding that is mediocre at representing any single one of them, so it ranks lower than it should for queries that are actually about content it contains.
• Context starvation: a 100-token chunk might contain the exact sentence a query needs, but without the paragraph around it the model can't resolve pronouns, table headers, or the section title that gives the sentence meaning.
• Retrieval fragmentation: a single logical idea, a code example plus its explanation, a table plus its caption, gets split across two or three chunks, and unless your top-k is generous enough to pull all of them back together, the model gets half the picture.

None of this is fixed by a bigger embedding model or a fancier reranker. Those components can only rerank or interpret what the chunker handed them. If the chunk boundaries destroy the information, nothing downstream recovers it.

How to choose a starting rag chunk size

Skip the temptation to search for "the correct chunk size." There isn't one, because it depends on your content type, your embedding model's effective context window, and your query patterns. What does exist is a sane default you can tune from.

For dense prose, product docs, knowledge base articles, blog content, a reasonable starting point is 300 to 500 tokens per chunk with 10 to 20 percent overlap. That range tends to keep one coherent idea per chunk: a paragraph or two, or a subsection with its heading. For technical or reference content with code, API docs, config examples, you generally want larger chunks (500 to 800 tokens) because splitting a code block from its explanation is worse than a slightly diluted embedding. For chat transcripts, support tickets, or Q&A pairs, chunk per turn or per exchange instead of a fixed token count, since the natural unit of meaning is already delimited by the conversation structure.

A few things to check before you commit to a number:

1. Your embedding model's practical sweet spot. Most embedding models are trained and evaluated on passages in the few-hundred-token range. Feeding them multi-thousand-token chunks pushes them outside where they perform best, even if the model technically accepts a longer input.
2. Your generation model's context budget. If you retrieve top-8 chunks at 500 tokens each, that's 4000 tokens of context before your prompt template and the user's question. Work backward from how much retrieved context you can actually afford per query.
3. How atomic your source content already is. If your docs are already broken into short, well-scoped sections, don't fight that structure with a token-count splitter that ignores it. Structure-aware chunking beats fixed-size chunking whenever the structure is reliable.

Start with a default, run it against real queries, and adjust based on what you observe, not intuition.

Chunk overlap: what it buys you and what it costs

Overlap means adjacent chunks share a slice of text at their boundary, so if chunk 1 ends mid-sentence, chunk 2 repeats enough of the tail of chunk 1 to preserve the meaning. This solves the boundary problem where an important fact happens to land right at a chunk split and no single chunk contains it whole.

The tradeoff is storage and noise. More overlap means more total chunks for the same source text, more vectors to store, more compute at query time, and a higher chance that near-duplicate chunks both show up in your top-k, crowding out other genuinely distinct results.

A working rule: 10 to 20 percent of chunk size is enough overlap to catch most boundary-split facts without meaningfully bloating your index. For a 400-token chunk, that's 40 to 80 tokens of shared text. Push overlap toward 25 to 30 percent only if you've measured specific failures where facts near chunk boundaries were getting missed and smaller overlap didn't fix it.

Overlap is not a substitute for correct chunk boundaries. If you're relying on overlap to paper over a splitter that cuts through the middle of tables, code blocks, or list items, fix the splitter instead of cranking overlap higher.

Chunking strategies worth knowing

Fixed-size splitting cuts text every N tokens or characters, with no awareness of sentence or paragraph boundaries. It's fast and predictable but the most likely to slice through meaning. Only use it as a last resort or on genuinely unstructured text.

Recursive character splitting tries a hierarchy of separators, paragraph breaks first, then sentences, then words, falling back to a hard character cut only if nothing else gets the chunk under the size limit. This is the most common production default because it respects natural text boundaries while still guaranteeing a size ceiling.

Sentence-window chunking indexes individual sentences for precise similarity matching, but retrieves a window of surrounding sentences (the sentence before and after, or a fixed paragraph) at query time. This gives you tight embeddings for matching plus enough context for the model to actually use the result.

Semantic chunking uses an embedding model to detect where topic shifts happen in the text and splits there instead of at a fixed size. It produces more coherent chunks but costs extra embedding calls at ingest time and needs its own similarity threshold tuned per corpus.

Hierarchical or parent-child chunking indexes small chunks for retrieval precision, but each small chunk stores a pointer back to a larger parent chunk (a full section or document). When a small chunk matches a query, you fetch its parent for generation. This gets you the best of both worlds: precise matching, generous context, at the cost of a more complex retrieval step.

Here is a working example using LangChain's recursive splitter, tuned with the defaults discussed above:

from langchain_text_splitters import RecursiveCharacterTextSplitter
import tiktoken

encoding = tiktoken.get_encoding("cl100k_base")

def token_length(text: str) -> int:
    return len(encoding.encode(text))

splitter = RecursiveCharacterTextSplitter(
    chunk_size=400,
    chunk_overlap=60,
    length_function=token_length,
    separators=["\n\n", "\n", ". ", " ", ""],
)

with open("docs/product-guide.md") as f:
    raw_text = f.read()

chunks = splitter.split_text(raw_text)
print(f"Produced {len(chunks)} chunks")
print(f"Average chunk length: {sum(token_length(c) for c in chunks) / len(chunks):.0f} tokens")

And a hierarchical parent-child setup using LlamaIndex, which stores small chunks for search but resolves back to the full node for context:

from llama_index.core.node_parser import HierarchicalNodeParser
from llama_index.core import Document

node_parser = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[2048, 512, 128],
)

document = Document(text=raw_text)
nodes = node_parser.get_nodes_from_documents([document])

leaf_nodes = [n for n in nodes if not n.child_nodes]
print(f"Leaf nodes for retrieval: {len(leaf_nodes)}")

If you're rolling your own pipeline without a framework, a simple sentence-aware splitter with overlap looks like this:

import re

def chunk_text(text, max_tokens=400, overlap_tokens=60, token_fn=token_length):
    sentences = re.split(r"(?<=[.!?])\s+", text.strip())
    chunks = []
    current = []
    current_len = 0

    for sentence in sentences:
        sentence_len = token_fn(sentence)
        if current_len + sentence_len > max_tokens and current:
            chunks.append(" ".join(current))
            overlap_sentences = []
            overlap_len = 0
            for s in reversed(current):
                s_len = token_fn(s)
                if overlap_len + s_len > overlap_tokens:
                    break
                overlap_sentences.insert(0, s)
                overlap_len += s_len
            current = overlap_sentences
            current_len = overlap_len

        current.append(sentence)
        current_len += sentence_len

    if current:
        chunks.append(" ".join(current))

    return chunks

This keeps splits on sentence boundaries, respects your token budget, and carries a controlled amount of context forward into the next chunk instead of an arbitrary character cut.

Measuring the tradeoff instead of guessing

Chunk size tuning only means something if you can measure retrieval quality before and after a change. Build a small evaluation set: 20 to 50 real or realistic queries paired with the chunk (or document) that should be retrieved for each. This does not need to be elaborate, a spreadsheet or a JSON file mapping query to expected source is enough to start.

eval_set = [
    {"query": "how do I rotate api keys", "expected_source": "docs/security.md#key-rotation"},
    {"query": "refund policy for annual plans", "expected_source": "docs/billing.md#refunds"},
]

def evaluate_retrieval(retriever, eval_set, k=5):
    hits = 0
    for item in eval_set:
        results = retriever.search(item["query"], top_k=k)
        retrieved_sources = [r.metadata["source"] for r in results]
        if item["expected_source"] in retrieved_sources:
            hits += 1
    return hits / len(eval_set)

Run this against a few chunk size and overlap combinations (say 200/40, 400/60, 800/100) and track recall@k, whether the right chunk showed up at all in your top-k, alongside mean rank of the correct chunk, which tells you whether it's showing up first or barely scraping into the results. If recall improves as chunk size grows but mean rank gets worse, that's a sign chunks are getting diluted even as they happen to contain the right text somewhere inside them.

Also track chunk count and index size at each setting. A configuration that improves recall by two points but doubles your vector count and query latency might not be worth it in production, especially if you're paying per-query on a hosted vector database like Pinecone, Weaviate, or a managed pgvector instance.

Don't skip the qualitative pass either. Pull the actual chunks returned for a handful of queries and read them. Automated recall metrics won't tell you that a chunk is technically the right document but missing the one sentence with the actual number the user asked for, that only shows up when you read the retrieved text yourself.

Common mistakes when tuning chunk size

Using one chunk size for a mixed corpus. Docs, code, and support transcripts have different natural units of meaning. Route content type to content-appropriate chunking rules instead of running everything through the same splitter config.

Tuning against embedding similarity scores instead of end-task accuracy. A chunk size that maximizes cosine similarity scores isn't necessarily the one that gets the correct answer generated. Always evaluate against whether the model produced a correct, well-grounded answer, not just whether retrieval scores look good in isolation.

Ignoring the generation side of the budget. If you tune chunk size purely for retrieval precision without checking how many chunks fit in your prompt alongside the system instructions and conversation history, you can end up truncating context at generation time even though retrieval worked fine.

Re-chunking without re-indexing everything. Chunk size changes require a full re-embed and re-index of your corpus. Partial updates leave you with inconsistent chunk boundaries across your index, which makes debugging retrieval issues much harder because you can't tell if a bad result is a chunking problem or a stale-data problem.

Never revisiting the setting. Chunk size that worked for your initial 200-document corpus may not hold at 20,000 documents with different content mixes. Re-run your eval set periodically, especially after a significant content or embedding model change.

FAQ

What is a good default rag chunk size to start with? For general prose content, 300 to 500 tokens with 10 to 20 percent overlap is a reasonable starting point. Adjust from there based on your embedding model's characteristics and your evaluation results, not a fixed rule.

Does a bigger embedding model mean I can use bigger chunks? Not directly. Context window size and retrieval quality are different things. Even embedding models with large input limits tend to perform best on passages in the few-hundred-token range, because that's typically where they were trained and evaluated most heavily. Test your specific model rather than assuming a larger window means better results at larger chunk sizes.

Should I use the same chunk size for every document type in my corpus? No. Prose, code, tables, and chat logs have different natural units of meaning. A splitter tuned for paragraphs will mishandle code blocks and vice versa. Route different content types through chunking rules suited to their structure.

How much does chunk overlap actually help? It mainly prevents facts from being lost when they happen to land at a chunk boundary. 10 to 20 percent overlap catches most of these cases. Overlap beyond that mostly adds index size and near-duplicate results without meaningfully improving recall, unless you have specific evidence of boundary-related misses.

Is semantic chunking always better than fixed-size or recursive splitting? Not always. Semantic chunking produces more topically coherent chunks but adds ingest-time cost and an extra tuning parameter, the similarity threshold for splitting. Recursive character splitting with sensible separators is a strong default for most corpora, and semantic chunking is worth the extra complexity mainly when your content mixes topics within sections in ways structural splitting can't handle.

How do I know if my chunk size is actually the bottleneck versus my embedding model or reranker? Build a small eval set of query-to-expected-chunk pairs and test the same embedding model and reranker across a few chunk size configurations. If recall changes meaningfully as you vary chunk size alone, chunking is a real lever for you. If recall stays flat across configurations, look at your embedding model choice or add a reranking step instead.

Do I need to re-embed my whole corpus every time I change chunk size? Yes. Chunk boundaries define the vectors in your index, so changing chunk size or overlap means every chunk is different and needs to be re-embedded and re-indexed. Plan chunk size experiments on a representative subset of your corpus first, then commit to a full re-index only after you've validated the setting with your evaluation set.