Tuning RAG Chunk Overlap

RAG chunk overlap is the number of tokens (or characters) you repeat between consecutive chunks when you split a document before embedding it. Get it too low and you slice sentences and arguments in half, so the chunk that answers a question loses the context that made it retrievable. Get it too high and every chunk in your index carries duplicate text, which bloats storage, slows search, and lets near-identical chunks crowd out genuinely different ones in your top-k results. This guide walks through how overlap actually affects retrieval, gives you runnable code to test different settings, and ends with defaults you can start from for common document types.

What Chunk Overlap Actually Does

When you split a 2000-token document into 400-token chunks with zero overlap, chunk 1 covers tokens 0-400, chunk 2 covers 400-800, and so on. Any idea that happens to straddle token 400, say a sentence that starts at token 390 and ends at token 420, gets torn in two. Neither chunk contains the whole sentence. When that sentence is the one the answer depends on, your retriever either misses it entirely or retrieves half of it and your generator hallucinates the rest.

Overlap fixes this by letting chunk 2 start earlier, say at token 300 instead of 400. Now the 300-400 range appears in both chunks, so a sentence that crosses the boundary lives intact in at least one of them. The tradeoff is that you now store and embed that 100-token span twice. Multiply that across a 500-page manual and you can end up with 20-30% more chunks than a no-overlap split, each one costing embedding compute and vector storage.

This is the whole game with rag chunk overlap: you are trading index size and redundancy against the chance that any given fact survives a chunk boundary intact.

Why Getting RAG Chunk Overlap Wrong Breaks Retrieval

Overlap set too low shows up as a specific failure pattern: your RAG system answers confidently but wrong, and when you trace it back, the retrieved chunk contains the first half of a paragraph but not the sentence with the actual number, date, or clause. This is common in contracts, financial reports, and technical specs where a single sentence carries the load-bearing fact and it happens to sit near a chunk edge.

Overlap set too high shows up differently: your top-k retrieval results are near-duplicates of each other. If you ask for the top 5 chunks and overlap is 50% of chunk size, you might get 5 chunks that together cover only 3 chunks' worth of unique text, because two pairs are mostly the same content shifted by a few sentences. This wastes your context window budget and, worse, it can push out a chunk that actually contains different information but ranked slightly lower.

There's a subtler problem too: high overlap changes your embedding distribution. Near-duplicate chunks cluster tightly in vector space, which can bias cosine similarity scores upward for that region of the document. If one section of your corpus has small chunks with heavy overlap and another has larger chunks with none, you've built an index where similarity scores aren't comparable across sections, and your ranking quality degrades in ways that are hard to debug because nothing is throwing an error.

Picking a Starting Point for RAG Chunk Overlap

Before you tune anything, pick two numbers: chunk size and overlap. A common starting rule is overlap equal to 10-20% of chunk size. For a 512-token chunk, that's 50-100 tokens of overlap. This is a starting point, not a target, because the right number depends on how information is distributed in your source documents.

Ask yourself three questions about your corpus:

1. How long is a typical "unit of meaning"? A FAQ answer might be 2-3 sentences. A contract clause might be a full paragraph. A code function might be 30-80 lines. Your chunk size should comfortably fit one unit of meaning, and your overlap should be large enough to catch a unit that starts just before a boundary.
2. How dense is the information? Dense technical text (API references, legal clauses) needs more overlap per token because losing a few words changes meaning. Narrative text (blog posts, transcripts) tolerates less overlap because meaning is redundant across sentences.
3. How big is your corpus? If you have 50 documents, a slightly wasteful overlap costs you nothing. If you have 5 million documents, every extra percentage point of overlap is real infrastructure cost, and you should tune it down and compensate with smarter splitting (see the semantic chunking section below).

Token-Based Chunking with Overlap

Here's a minimal, dependency-light way to chunk text with a token-aware overlap using tiktoken. This keeps chunk size and overlap in token units, which matters because embedding models and LLM context windows are billed and limited in tokens, not characters.

import tiktoken

def chunk_text(text, chunk_size=400, overlap=80, encoding_name="cl100k_base"):
    enc = tiktoken.get_encoding(encoding_name)
    tokens = enc.encode(text)
    chunks = []
    start = 0
    step = chunk_size - overlap
    if step <= 0:
        raise ValueError("overlap must be smaller than chunk_size")

    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        chunk_tokens = tokens[start:end]
        chunks.append(enc.decode(chunk_tokens))
        if end == len(tokens):
            break
        start += step

    return chunks

with open("manual.txt") as f:
    text = f.read()

chunks = chunk_text(text, chunk_size=400, overlap=80)
print(f"produced {len(chunks)} chunks")
print(f"chunk 0 tail:\n{chunks[0][-200:]}\n")
print(f"chunk 1 head:\n{chunks[1][:200]}")

Run this on a document and inspect the printed tail/head pair. You should see the same sentence or two appearing at the end of chunk 0 and the start of chunk 1. If you don't see any repeated text, your overlap is too small relative to sentence length in that document.

If you're already using LangChain, RecursiveCharacterTextSplitter gives you overlap for free and tries to split on paragraph and sentence boundaries first, which reduces how often overlap needs to rescue a torn sentence:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1600,       # characters, roughly 400 tokens for English prose
    chunk_overlap=320,     # 20% of chunk_size
    separators=["\n\n", "\n", ". ", " ", ""],
)

chunks = splitter.split_text(text)

Notice the separator list is ordered from largest structural break to smallest. The splitter tries to cut on paragraph breaks first, and only falls back to cutting mid-sentence if a paragraph is too long to fit in one chunk. This matters more than the overlap number in a lot of cases: a good separator hierarchy means overlap has to do less rescue work.

Semantic Chunking as an Alternative to Fixed Overlap

Fixed-size chunking with overlap is a blunt instrument: it doesn't know where ideas actually start and end, so it uses redundancy to hedge against cutting in the wrong place. Semantic chunking tries to cut at the right place instead, using embedding similarity between adjacent sentences to detect topic shifts.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

def semantic_chunks(sentences, similarity_threshold=0.55, max_sentences=12):
    embeddings = model.encode(sentences, normalize_embeddings=True)
    chunks = []
    current = [sentences[0]]

    for i in range(1, len(sentences)):
        sim = float(np.dot(embeddings[i - 1], embeddings[i]))
        breaks_here = sim < similarity_threshold or len(current) >= max_sentences
        if breaks_here:
            chunks.append(" ".join(current))
            current = [sentences[i]]
        else:
            current.append(sentences[i])

    if current:
        chunks.append(" ".join(current))
    return chunks

When sentence-to-sentence similarity drops below the threshold, that's treated as a topic boundary and a new chunk starts. This approach doesn't need overlap in the traditional sense because it isn't cutting arbitrarily, but in practice teams still add a small 1-2 sentence overlap at each boundary as insurance against a bad similarity read. Semantic chunking costs more at index time (you're running an embedding pass just to decide where to cut, then embedding again for retrieval), so it's worth it mainly for corpora where meaning genuinely follows topic shifts, like long-form articles or meeting transcripts, and less worth it for uniformly structured data like tables or code.

Measuring the Effect: A Simple Eval Harness

Don't guess at overlap. Build a small labeled set of question-answer pairs where you know which source passage contains the answer, then measure retrieval recall at different overlap settings.

import json

def load_eval_set(path):
    # each line: {"question": "...", "answer_span": "exact text that must be retrieved"}
    with open(path) as f:
        return [json.loads(line) for line in f]

def recall_at_k(eval_set, retriever, k=5):
    hits = 0
    for item in eval_set:
        results = retriever.search(item["question"], top_k=k)
        found = any(item["answer_span"] in r["text"] for r in results)
        hits += int(found)
    return hits / len(eval_set)

def sweep_overlap(text, eval_set, build_index_fn, retriever_fn,
                   chunk_size=400, overlaps=(0, 40, 80, 120, 160)):
    results = {}
    for overlap in overlaps:
        chunks = chunk_text(text, chunk_size=chunk_size, overlap=overlap)
        index = build_index_fn(chunks)
        retriever = retriever_fn(index)
        results[overlap] = recall_at_k(eval_set, retriever, k=5)
    return results

Run sweep_overlap across a few overlap values and plot recall against overlap. In most corpora the curve rises quickly, plateaus, and then either flattens or slightly drops as duplicate chunks start displacing unique ones in the top-k. The overlap value at the start of the plateau, not the peak, is usually the right choice, because it gives you the recall benefit without paying the storage and redundancy cost of pushing further right on the curve.

If you don't have a labeled eval set yet, build one quickly by sampling 30-50 real or realistic user questions, manually finding the exact source passage that answers each, and copying that span verbatim into answer_span. This is an afternoon of work and it will tell you more about your chunking strategy than any amount of manual inspection.

Overlap by Document Type

Different document types need different overlap strategies because information density and structure differ so much:

• FAQs and short-form Q&A: chunk per Q&A pair, little or no overlap needed since each pair is already a complete unit.
• Technical documentation and API references: medium overlap (15-25% of chunk size), because code examples and parameter descriptions often reference the sentence before them.
• Legal contracts and policies: higher overlap (25-30%) or, better, chunk by clause number if the document has one, since a single sentence can carry binding meaning and losing half of it is costly.
• Long-form articles and blog content: lower overlap (10-15%) works fine because narrative prose restates its point across sentences, so a clean cut rarely loses the core idea.
• Source code: chunk by function or class boundary using an AST parser instead of token count, and use overlap only for imports or class-level context that a function depends on. Naive token-based overlap on code tends to duplicate boilerplate rather than meaningful context.
• Transcripts and meeting notes: chunk by speaker turn or topic segment (semantic chunking works well here), with a small 1-2 turn overlap so a response makes sense without needing to look back.

Common Mistakes

Using character count when you mean token count. A 500-character overlap is not the same as a 500-token overlap, and if your chunk size is defined in tokens but your overlap is defined in characters, the ratio between them silently drifts across languages and text styles that vary in characters-per-token.

Setting overlap as a fixed number instead of a percentage. A fixed 100-token overlap is generous for a 200-token chunk (50%) and stingy for a 2000-token chunk (5%). Define overlap relative to chunk size so it scales when you change chunk size later.

Never re-testing after changing the embedding model. Overlap tuning is somewhat model-specific because different embedding models weight local versus global context differently. If you swap embedding models, rerun your recall sweep before assuming your old overlap value still holds.

Ignoring overlap's effect on reranking cost. If you rerank the top-k results with a cross-encoder before sending them to the generator, near-duplicate chunks from high overlap waste reranker calls on redundant text. This is invisible in recall metrics but shows up as latency and cost.

Treating overlap as a fix for bad chunk boundaries. If your splitter is cutting mid-table or mid-code-block, more overlap won't help because the real problem is the separator hierarchy, not the overlap amount. Fix the splitter's boundary logic first, then tune overlap for the remaining edge cases.

FAQ

What is a good default RAG chunk overlap to start with? Start with overlap equal to 15-20% of your chunk size, for example 60-80 tokens for a 400-token chunk. Treat this as a baseline to test against your own eval set rather than a final answer, since the right value shifts with document type and embedding model.

Does higher chunk overlap always improve retrieval quality? No. Recall against a labeled eval set typically rises with overlap up to a point, then plateaus or drops slightly as near-duplicate chunks start competing for the same top-k slots. Past the plateau you're mostly paying storage and latency cost for no retrieval benefit.

Should I use the same overlap for every document type in one index? Not if your corpus mixes structurally different content, like FAQs and legal contracts in the same index. Chunk each document type with settings suited to it, then embed everything into the same vector store. The overlap value doesn't need to be uniform across the index.

Can semantic chunking replace overlap entirely? Mostly, but not completely. Semantic chunking reduces how often you need overlap to rescue a badly placed cut, since it tries to cut at real topic boundaries. Most teams still keep a small 1-2 sentence overlap at each boundary as a cheap safety margin against a bad similarity read.

How do I know if my current overlap setting is too low? Trace a sample of wrong or incomplete answers back to the retrieved chunks. If you repeatedly find the retrieved chunk contains the first half of a sentence or paragraph but not the part with the actual fact, that's the signature of overlap set too low relative to your document's sentence and paragraph lengths.

Does chunk overlap affect embedding cost? Yes, directly. Overlap increases the total token count you embed and store, since the overlapping span is embedded once per chunk it appears in. A 20% overlap on a large corpus can add roughly 20-25% more chunks than a zero-overlap split of the same chunk size, so budget for that when estimating embedding and vector storage cost.