RAG

Chunk Size and Overlap in RAG: Finding the Sweet Spot

Ira Menon · May 13, 2026 · 14 min read

Every RAG pipeline eventually runs into the same wall. Retrieval "works" — the vector database returns results, the LLM generates an answer — but the answers are subtly wrong, missing context, or repeating the same paragraph twice. Nine times out of ten, when I trace the problem back to its root, it's not the embedding model or the LLM at all. It's chunking. Specifically, it's a badly chosen combination of chunk size overlap rag settings that nobody revisited after the first prototype worked "well enough." Chunk size and overlap are the two knobs that decide what actually ends up inside your vector index, and they quietly determine the ceiling on your retrieval quality no matter how good your model is downstream. This article walks through how to think about both, with working code, so you can stop guessing and start measuring.

Why Chunking Matters More Than People Think

When you build a RAG system, you're not indexing documents — you're indexing chunks. The chunk is the atomic unit of retrieval. If a chunk is too large, it dilutes the embedding with irrelevant text, and the vector similarity search becomes less precise because the vector now represents an average of several ideas instead of one coherent one. If a chunk is too small, you lose context: a sentence like "This applies only to Enterprise plan customers" means nothing without the preceding paragraph that explains what "this" refers to.

The tension is fundamental: chunk size trades off *precision* (small chunks are easier to match precisely to a query) against *context* (large chunks preserve more surrounding information so the LLM isn't working with a fragment). Overlap is the patch for one specific failure mode created by chunking at all — the fact that important information often sits right at a chunk boundary and gets split in half.

I've seen teams spend weeks tuning prompts and swapping embedding models when the actual fix was changing chunk_size from 2000 to 500 and adding a sensible overlap. Chunking is unglamorous, so it gets skipped over, but it's frequently the single highest-leverage change you can make in an existing RAG system.

What Chunk Size Actually Controls

Chunk size is usually measured in tokens or characters, and it controls three things simultaneously:

Embedding fidelity — most embedding models (OpenAI's text-embedding-3-small, Cohere's embed models, open-source models like bge-large) were trained and evaluated on relatively short passages, typically in the 256–512 token range. Push far beyond that and the embedding starts to represent a blurry average of multiple topics rather than one specific idea.
Retrieval granularity — smaller chunks mean you can retrieve more precisely targeted pieces of text. If your knowledge base has one paragraph that directly answers the query, a 300-token chunk finds it. A 3000-token chunk finds the whole section it lives in, which may drown the one useful paragraph in four unrelated ones.
Context budget — every chunk you retrieve eats into your LLM's context window. If you retrieve the top 5 chunks and each one is 2000 tokens, you've spent 10,000 tokens before the model has even started reasoning. That's expensive and can push out room for conversation history or system instructions.

Here's a simple way to see the effect directly. Take the same document, chunk it at two different sizes, and look at what a query actually retrieves:

from langchain_text_splitters import RecursiveCharacterTextSplitter

document = """
Refund Policy: Customers may request a refund within 30 days of purchase
if the product was not used more than twice. Digital products, including
course access and downloadable materials, are non-refundable once the
first module has been accessed. Enterprise customers on annual contracts
are subject to a separate refund schedule outlined in their Master
Service Agreement, which typically allows a pro-rated refund within the
first 60 days. Refund requests must be submitted through the billing
portal, not via email, to ensure proper tracking and audit compliance.
"""

small_splitter = RecursiveCharacterTextSplitter(chunk_size=120, chunk_overlap=0)
large_splitter = RecursiveCharacterTextSplitter(chunk_size=600, chunk_overlap=0)

small_chunks = small_splitter.split_text(document)
large_chunks = large_splitter.split_text(document)

print(f"Small chunks: {len(small_chunks)}")
for c in small_chunks:
    print(" -", c.strip().replace("\n", " "))

print(f"\nLarge chunks: {len(large_chunks)}")
for c in large_chunks:
    print(" -", c.strip().replace("\n", " "))

Run that and you'll notice the small chunks isolate the Enterprise refund rule cleanly, while the large chunk merges it with the general consumer refund policy. If a user asks "what's the refund window for enterprise customers," the small chunk is a near-perfect embedding match. The large chunk is a diluted match that competes with dozens of other general-policy chunks in your index.

The Case for Overlap

Overlap exists to solve one problem: boundary splitting. No matter how you choose your chunk size, some sentence, table row, or logical unit will land exactly on the cut line. Without overlap, that unit gets sliced into two chunks, and each half loses the meaning the other half provided.

Consider this text:

...the migration script will drop the legacy `user_sessions` table.
Before running it in production, make sure you have exported all
session data using the `export_sessions.py` utility, since this
operation cannot be undone once the table is dropped.

If your chunk boundary lands right after "make sure you have exported all," you get two chunks:

Chunk A ends with: "...before running it in production, make sure you have exported all"
Chunk B starts with: "session data using the export_sessions.py utility, since this operation cannot be undone..."

Neither chunk alone tells the reader (or the retrieval system) what needs to be exported, or why it matters. Overlap re-includes the tail of chunk A at the head of chunk B (or vice versa), so the critical sentence survives intact in at least one chunk.

A typical overlap is 10–20% of the chunk size. If your chunk size is 500 tokens, an overlap of 50–100 tokens is a reasonable starting point. Here's the same document chunked with and without overlap so you can compare directly:

from langchain_text_splitters import RecursiveCharacterTextSplitter

text = """The migration script will drop the legacy user_sessions table.
Before running it in production, make sure you have exported all
session data using the export_sessions.py utility, since this
operation cannot be undone once the table is dropped. After the
export completes, verify the backup file size matches the expected
row count before proceeding with the migration."""

no_overlap = RecursiveCharacterTextSplitter(chunk_size=150, chunk_overlap=0)
with_overlap = RecursiveCharacterTextSplitter(chunk_size=150, chunk_overlap=40)

print("No overlap:")
for c in no_overlap.split_text(text):
    print(" -", repr(c.strip()))

print("\nWith overlap:")
for c in with_overlap.split_text(text):
    print(" -", repr(c.strip()))

The overlapping version costs you extra storage and slightly more embedding compute, since the same sentence gets embedded more than once across adjacent chunks. That's the trade-off: overlap buys context continuity at the cost of index size and some redundant retrieval. In practice, for most production systems that cost is trivial compared to the accuracy gain.

Fixed-Size vs Semantic Chunking

There are two broad families of chunking strategy, and chunk size behaves differently in each.

Fixed-size (or recursive character) chunking splits text based on a target length, trying to break at natural boundaries like paragraphs or sentences when possible, but ultimately enforcing a hard size limit. This is what RecursiveCharacterTextSplitter and similar utilities do. It's predictable, fast, and works on any text, but it's blind to meaning — it doesn't know or care whether a chunk boundary falls in the middle of a coherent idea.

Semantic chunking instead tries to split text at points where the *meaning* shifts, using sentence embeddings to detect topic boundaries. The chunk size becomes a byproduct of the content rather than a fixed target — a section that stays on one topic for 800 tokens becomes one chunk; a section that changes topic every 100 tokens becomes several small chunks.

# Rough sketch of the idea behind semantic chunking
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

def semantic_chunk(sentences, threshold=0.55):
    embeddings = model.encode(sentences)
    chunks = []
    current_chunk = [sentences[0]]

    for i in range(1, len(sentences)):
        sim = np.dot(embeddings[i], embeddings[i - 1]) / (
            np.linalg.norm(embeddings[i]) * np.linalg.norm(embeddings[i - 1])
        )
        if sim < threshold:
            chunks.append(" ".join(current_chunk))
            current_chunk = [sentences[i]]
        else:
            current_chunk.append(sentences[i])

    chunks.append(" ".join(current_chunk))
    return chunks

Semantic chunking tends to produce better retrieval quality on documents with mixed content — think a long onboarding doc that covers billing, permissions, and API keys in sequence. But it's slower to build (you're running embeddings just to decide where to cut), harder to reason about (chunk sizes become unpredictable, which complicates your context budget), and it still needs a fallback maximum size, because a document that never shifts topic will produce one giant chunk that blows your embedding model's effective range.

My default recommendation for most teams: start with fixed-size recursive chunking with a sensible overlap. It's boring, but boring is debuggable. Move to semantic chunking only after you've measured that boundary-splitting is actually your bottleneck — not before.

Structure-Aware Chunking

Plain text is only part of the picture. If your source documents have structure — markdown headers, HTML sections, code blocks, table rows — ignoring that structure while chunking throws away free information.

from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

headers_to_split_on = [
    ("#", "h1"),
    ("##", "h2"),
    ("###", "h3"),
]

markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
header_chunks = markdown_splitter.split_text(markdown_document)

# Then apply size-based splitting within each header section
final_splitter = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=50)
final_chunks = []
for doc in header_chunks:
    for sub_chunk in final_splitter.split_text(doc.page_content):
        final_chunks.append({
            "text": sub_chunk,
            "metadata": doc.metadata,  # carries h1/h2/h3 context forward
        })

This two-pass approach — split by structure first, then by size within each structural unit — gives you the best of both. You never merge unrelated sections (chunking respects the document's own organization), and you never end up with a single chunk that's an entire 5,000-word chapter. The header metadata you attach here is also useful later: you can inject "Section: Billing > Refunds > Enterprise" into the prompt alongside the chunk text, giving the LLM context that the raw chunk alone wouldn't carry.

Code is a special case worth calling out separately. Chunking source code with a plain character splitter is a common mistake — it happily cuts a function in half. Use a language-aware splitter that respects function and class boundaries instead:

from langchain_text_splitters import RecursiveCharacterTextSplitter, Language

python_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON, chunk_size=500, chunk_overlap=50
)
code_chunks = python_splitter.split_text(source_code)

Choosing a Starting Point

There's no universal "correct" chunk size, but here are starting points I actually use, based on document type and the kind of question users tend to ask:

FAQ or support docs (short, self-contained answers): chunk size 200–400 tokens, overlap 20–40 tokens. Each chunk should roughly correspond to one Q&A pair or one policy statement.
Technical documentation / API references: chunk size 400–800 tokens, overlap 50–100 tokens. You want enough room for a code example plus its explanation to stay together.
Long-form narrative content (reports, contracts, books): chunk size 800–1500 tokens, overlap 100–200 tokens. These documents build meaning across paragraphs, so cutting too aggressively destroys context.
Legal or compliance documents: chunk size 500–1000 tokens, overlap 100–150 tokens, ideally combined with structure-aware splitting on clause or section numbers. Precision matters enormously here — you do not want an LLM answering a compliance question from a half-clause.

These are starting points, not settled answers. The only way to know whether they're right for your data is to test them against real queries from real users, which brings us to the part most teams skip entirely.

Measuring What Actually Works: Retrieval Evaluation

You cannot tune chunk size and overlap by vibes. You need a small evaluation set: a list of representative questions paired with the chunk(s) that should be retrieved to answer them correctly. Even 20–30 examples is enough to start catching regressions.

eval_set = [
    {
        "question": "What is the refund window for enterprise customers?",
        "expected_source": "refund_policy.md#enterprise",
    },
    {
        "question": "How do I export session data before running the migration?",
        "expected_source": "migration_guide.md#pre-migration-steps",
    },
    # ...
]

def evaluate_retrieval(retriever, eval_set, k=5):
    hits = 0
    for item in eval_set:
        results = retriever.retrieve(item["question"], top_k=k)
        retrieved_sources = [r.metadata.get("source") for r in results]
        if item["expected_source"] in retrieved_sources:
            hits += 1
    recall_at_k = hits / len(eval_set)
    print(f"Recall@{k}: {recall_at_k:.2%}")
    return recall_at_k

Run this same evaluation across a grid of chunk size and overlap combinations, and you'll get a real answer instead of a guess:

configs = [
    {"chunk_size": 256, "chunk_overlap": 25},
    {"chunk_size": 512, "chunk_overlap": 50},
    {"chunk_size": 512, "chunk_overlap": 100},
    {"chunk_size": 1024, "chunk_overlap": 100},
]

for config in configs:
    splitter = RecursiveCharacterTextSplitter(**config)
    chunks = splitter.split_text(full_corpus)
    index = build_vector_index(chunks)  # your embedding + index step
    retriever = index.as_retriever()
    print(f"\nConfig: {config}")
    evaluate_retrieval(retriever, eval_set)

What you're looking for is the point of diminishing returns — where increasing chunk size stops improving recall and starts hurting it because chunks are getting noisy, or where increasing overlap stops fixing boundary issues and starts just wasting index space. In my experience this curve is not subtle once you plot it; there's usually a clear peak, and it's rarely the config the team started with by default.

Common Mistakes I See Repeatedly

Copying a chunk size from a blog post or tutorial without testing it on your own data. A chunk size tuned for Wikipedia-style prose behaves differently on dense legal text or terse Slack-message-style support tickets.
Setting overlap to zero to save index space. This is the single most common cause of "the answer is almost right but missing one crucial detail" bugs. Overlap is cheap; debugging silently truncated context is not.
Using the same chunk size for every document type in a mixed corpus. A 500-token chunk might be perfect for your FAQ but terrible for your 40-page product spec. Chunk by document type, not globally.
Ignoring chunk size when switching embedding models. Different embedding models have different effective context ranges. If you swap from an older 512-token-limit model to a longer-context one, your old chunk size choice is no longer validated for the new model — retest.
Forgetting to re-chunk after adding overlap. Overlap only matters if it's actually applied at retrieval time and at generation time consistently — I've seen pipelines where overlap was added to the chunking config but the vector store was never rebuilt, so nothing changed.
Not deduplicating overlapping content in the final prompt. When overlap is generous and top-k retrieval pulls in adjacent chunks, the LLM sometimes receives the same sentence twice in its context. This wastes tokens and can confuse the model into repeating itself. A light post-retrieval dedup pass (even a simple string similarity check between adjacent chunks) fixes this cheaply.

Putting It Together: A Sensible Default Pipeline

If you're starting a new RAG project today and want a defensible default rather than a perfectly tuned one, this is roughly what I'd ship on day one, then iterate from evaluation data:

from langchain_text_splitters import RecursiveCharacterTextSplitter

def build_chunks(document_text: str, doc_type: str = "general"):
    configs = {
        "faq": {"chunk_size": 300, "chunk_overlap": 30},
        "technical_docs": {"chunk_size": 600, "chunk_overlap": 75},
        "long_form": {"chunk_size": 1000, "chunk_overlap": 150},
        "general": {"chunk_size": 500, "chunk_overlap": 50},
    }
    config = configs.get(doc_type, configs["general"])
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=config["chunk_size"],
        chunk_overlap=config["chunk_overlap"],
        separators=["\n\n", "\n", ". ", " ", ""],
    )
    return splitter.split_text(document_text)

Notice the separators list: it tells the splitter to prefer breaking on paragraph boundaries first, then line breaks, then sentence boundaries, and only fall back to splitting mid-word as an absolute last resort. This one parameter does a lot of quiet work in keeping chunks semantically coherent even within a purely fixed-size approach.

Ship this, instrument retrieval with logging (store which chunks were retrieved for which query, and whether the user thumbs-upped or thumbs-downed the answer), and revisit your chunk size and overlap numbers monthly using real production queries rather than synthetic test questions. Chunking is not a one-time decision — it's a setting that should evolve as your document corpus and your user base evolve.

Closing Thoughts

Chunk size and overlap look like small implementation details, but they are two of the highest-leverage levers you have in a RAG system, and they're far cheaper to tune than swapping models or rewriting prompts. Start with sensible, document-type-aware defaults, add overlap generously rather than sparingly, respect document structure when you have it, and — most importantly — build a small evaluation set so you're tuning against real recall numbers instead of intuition. Once chunking is solid, everything downstream — retrieval, ranking, generation — gets noticeably easier to reason about. If you want to go deeper into how retrieval, embeddings, and generation fit together end to end, our Introduction to RAG course walks through the full pipeline from first principles, chunking included, with hands-on exercises you can run against your own documents.