RAG

RAG Chunking Strategies Compared: Fixed, Recursive, Semantic and Agentic

Pramod Dutta · May 13, 2026 · 15 min read

Every RAG pipeline eventually hits the same wall. Retrieval quality is flat, the LLM keeps hallucinating details that are technically "in the document," and no amount of prompt tweaking fixes it. Nine times out of ten, the real problem isn't the embedding model or the retriever — it's chunking. How you split your documents before you ever embed them determines the ceiling on everything downstream. Get chunking wrong and you're feeding the model fragments that cut sentences in half, split a table from its caption, or bury the one relevant paragraph inside four unrelated ones. Get it right and retrieval suddenly feels almost magical. This piece walks through the four chunking strategies you'll actually encounter in production — fixed-size, recursive, semantic, and agentic — with working code, honest trade-offs, and a framework for picking the right one for your data.

Why chunking is the most underrated part of RAG

Most tutorials spend paragraphs on vector databases and embedding model selection, then hand-wave chunking with a single RecursiveCharacterTextSplitter(chunk_size=1000) call and move on. That's backwards. The embedding model only ever sees what you hand it — if a chunk boundary slices a definition away from the term it defines, no amount of embedding sophistication recovers that lost context. rag chunking strategies are the first and most consequential decision in the entire pipeline, because every downstream step — embedding, indexing, retrieval, reranking, generation — operates on the units you created at this step. You cannot retrieve information that was never grouped together in the first place.

There's also a cost dimension people ignore. Chunk size directly affects your token spend at both indexing time and query time. Smaller chunks mean more vectors to store and more retrieval calls needed to reconstruct context; larger chunks mean each retrieved item drags in more irrelevant text that dilutes the LLM's attention and burns tokens in the context window. Chunking is a tuning knob for accuracy, latency, and cost simultaneously — which is exactly why it deserves more attention than it usually gets.

Fixed-size chunking: the baseline everyone starts with

Fixed-size chunking splits text into chunks of a set length — say 500 characters or 256 tokens — often with some overlap between consecutive chunks so context isn't lost at the boundary. It's the "hello world" of RAG chunking strategies, and for good reason: it's fast, deterministic, and requires no understanding of the document's structure.

def fixed_size_chunks(text, chunk_size=500, overlap=50):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks

document = open("product_manual.txt").read()
chunks = fixed_size_chunks(document, chunk_size=500, overlap=50)
print(f"Generated {len(chunks)} chunks")

The overlap parameter matters more than people think. Without it, a sentence that straddles the boundary between chunk 3 and chunk 4 gets truncated in both, and neither chunk is retrievable as a coherent unit. A 10-20% overlap (50 characters on a 500-character chunk) is a reasonable default, though you're still paying storage cost for duplicated text.

The problem with fixed-size chunking is obvious once you see it in practice: it doesn't care about meaning. It will cheerfully cut a bulleted list in half, split a code function from its docstring, or break a paragraph mid-sentence. For quick prototypes, internal tools, or homogeneous plain-text corpora where structure doesn't carry meaning, fixed-size chunking is perfectly adequate. For anything customer-facing or anything built on structured documents — technical docs, contracts, markdown files with headers — it's usually the wrong default, even though it's the one most tutorials ship with.

Recursive chunking: respecting structure without over-engineering

Recursive chunking is the pragmatic middle ground, and it's what most production RAG systems actually use as their default. Instead of blindly cutting at a character count, it tries a hierarchy of separators — paragraphs, then sentences, then words — and only falls back to a harder split when a chunk still exceeds the target size.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", " ", ""]
)

with open("onboarding_guide.md") as f:
    text = f.read()

chunks = splitter.split_text(text)
for i, chunk in enumerate(chunks[:3]):
    print(f"--- Chunk {i} ({len(chunk)} chars) ---")
    print(chunk)

The logic is straightforward: try splitting on double newlines first (paragraph boundaries). If a resulting piece is still too big, split that piece on single newlines. Still too big? Split on sentence boundaries. Still too big? Split on spaces. This means recursive chunking naturally respects the document's existing structure — paragraphs stay together when they fit, and only genuinely oversized blocks get force-split.

This is the strategy I reach for by default when building a RAG system for markdown docs, wikis, or general prose. It's cheap (no extra model calls, no embedding computation during the split itself), it's fast enough to run over tens of thousands of documents in minutes, and it produces noticeably better chunks than the fixed-size approach for almost zero extra engineering cost. The main tuning work is picking chunk_size for your domain — legal contracts might want 1500-character chunks to keep clauses intact, while FAQ-style content might do better with 300-400 character chunks per answer.

Recursive chunking still has a blind spot: it has no idea what the text *means*. Two consecutive paragraphs that are topically unrelated will get merged into one chunk if they fit under the size limit, and a single coherent idea that happens to span three paragraphs might get split apart. That's the gap semantic chunking tries to close.

Semantic chunking: splitting where the meaning actually changes

Semantic chunking uses embeddings to detect where the *topic* shifts, rather than relying on character counts or punctuation. The typical approach: split the document into sentences, embed each sentence (or a small sliding window of sentences), and measure the cosine distance between consecutive embeddings. When the distance spikes above a threshold, that's a signal the topic changed — and that's where you cut.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

def semantic_chunks(sentences, threshold=0.35):
    embeddings = model.encode(sentences)
    chunks, current_chunk = [], [sentences[0]]

    for i in range(1, len(sentences)):
        sim = np.dot(embeddings[i], embeddings[i - 1]) / (
            np.linalg.norm(embeddings[i]) * np.linalg.norm(embeddings[i - 1])
        )
        distance = 1 - sim
        if distance > threshold:
            chunks.append(" ".join(current_chunk))
            current_chunk = [sentences[i]]
        else:
            current_chunk.append(sentences[i])

    if current_chunk:
        chunks.append(" ".join(current_chunk))
    return chunks

sentences = [
    "The onboarding flow starts with email verification.",
    "Users receive a six-digit code valid for ten minutes.",
    "Pricing tiers are billed monthly or annually.",
    "The enterprise tier includes a dedicated account manager.",
]
result = semantic_chunks(sentences)
print(result)

In this toy example, the semantic splitter should catch the shift from "onboarding/verification" to "pricing/billing" and produce two chunks instead of blindly merging all four sentences because they fit a character budget. That's the entire value proposition: chunk boundaries align with topic boundaries, which means each chunk is more likely to be a complete, self-contained unit of meaning — which in turn means your retriever is more likely to return something the LLM can use without needing surrounding context it didn't retrieve.

The cost is real, though. Semantic chunking requires embedding every sentence (or window) at indexing time just to *decide where to cut* — before you've even created the chunks you'll embed for retrieval. For a 10,000-page knowledge base, that's a meaningfully larger compute and time bill than recursive chunking, which does zero model inference. The threshold is also finicky: set it too low and you get chunks the size of single sentences (too granular, retrieval noise goes up); set it too high and you're back to giant chunks that ignore topic shifts. In practice, teams tune this threshold empirically against a labeled eval set rather than picking a number a priori.

Semantic chunking earns its cost on heterogeneous, information-dense documents — long-form reports, meeting transcripts, research papers — where topics genuinely drift within a single document and structural cues (headers, paragraph breaks) don't reliably mark those transitions. For well-structured markdown with clear headers, recursive chunking usually gets you 90% of the benefit for a fraction of the cost.

There's a middle-ground variant worth knowing about too: instead of comparing every adjacent sentence pair, you compute embeddings for a sliding window of sentences (say, three at a time) and look for the biggest jumps in distance across the whole document, then cut at the top N jumps. This "breakpoint" approach is more robust to noisy single-sentence embeddings and tends to produce more consistent chunk sizes than the naive adjacent-pair method, at the cost of an extra pass over the similarity scores to find the actual breakpoints. Either way, plan on treating the threshold or breakpoint count as a hyperparameter you tune against your own eval set, not a constant you borrow from someone else's blog post.

Agentic chunking: letting an LLM decide the boundaries

Agentic chunking is the newest and most expensive strategy: instead of a rule or an embedding-distance heuristic, you ask an LLM to read the document and decide, semantically, where the natural boundaries are — often producing chunks structured as self-contained "propositions" or topic units, sometimes with generated summaries or metadata attached to each one.

import json
from anthropic import Anthropic

client = Anthropic()

CHUNKING_PROMPT = """You are splitting a document into self-contained chunks for a RAG system.
Each chunk should:
- Cover exactly one topic or idea
- Be understandable without reading other chunks
- Preserve any necessary context (e.g. restate what "it" refers to)

Return a JSON list of objects: {{"chunk": "...", "topic": "..."}}

Document:
{document}
"""

def agentic_chunk(document):
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=4096,
        messages=[{"role": "user", "content": CHUNKING_PROMPT.format(document=document)}],
    )
    return json.loads(response.content[0].text)

doc = open("refund_policy.txt").read()
chunks = agentic_chunk(doc)
for c in chunks:
    print(f"[{c['topic']}] {c['chunk'][:80]}...")

What makes this "agentic" rather than just "LLM-assisted" is that some implementations go further — the model doesn't just cut once, it can evaluate its own output, merge chunks that turned out too granular, re-split ones that are still doing too much, or even generate a one-line contextual summary prepended to each chunk (a pattern popularized as "contextual retrieval," where each chunk gets a short LLM-written blurb explaining how it relates to the whole document before embedding). This produces the highest-quality chunks of any strategy discussed here, because the LLM actually understands pronouns, cross-references, and implicit context in a way no embedding-distance heuristic can.

The catch is cost and latency, and it's not a small catch. Running every document through an LLM call (or several, if the agent iterates) at ingestion time is orders of magnitude more expensive than recursive or even semantic chunking. For a knowledge base that updates constantly, this can become the dominant cost line in your entire RAG budget. Agentic chunking makes sense for high-value, relatively static corpora — legal contracts, compliance documentation, a curated internal wiki that changes weekly, not hourly — where retrieval precision is worth paying for and the document set is small enough that re-chunking everything isn't a recurring tax.

A practical decision framework

Here's how I'd actually choose between these four rag chunking strategies on a real project, roughly in order of what to check first:

Is this a prototype or proof-of-concept? Use fixed-size or recursive chunking. Don't spend engineering time on semantic or agentic chunking before you've validated the product idea.
Is the source well-structured (markdown, HTML with headers, code with docstrings)? Recursive chunking with separators tuned to that structure (e.g. splitting on ## headers first) will outperform semantic chunking most of the time, for a fraction of the cost.
Is the source unstructured prose where topics drift mid-document (meeting notes, long articles, research papers)? Semantic chunking is worth the extra embedding cost.
Is retrieval precision worth a real dollar cost, and is the corpus small or slow-changing? Agentic chunking, possibly combined with LLM-generated chunk summaries, gives you the best quality ceiling.
Are you re-chunking on every update, or once? If your corpus updates hourly, the recurring cost of semantic or agentic chunking compounds fast — budget for it or fall back to recursive.

A pattern I use often in practice: don't commit to one strategy globally. A single RAG system can — and often should — use different chunking strategies for different document types. FAQ pages might get simple fixed-size chunks per Q&A pair. A markdown knowledge base gets recursive chunking keyed off header levels. A folder of long-form legal PDFs gets semantic or agentic chunking because the retrieval stakes are high and the update frequency is low.

Chunk size and overlap still matter, regardless of strategy

Whichever strategy you pick, two parameters need tuning against your actual data, not copied from a blog post: chunk size and overlap.

Too small a chunk size (under ~200 tokens) and you lose context — the retriever finds a matching sentence but the LLM can't tell what it's about without the surrounding paragraph.
Too large a chunk size (over ~1500 tokens) and you dilute relevance — the embedding for the chunk becomes an average of several ideas, so it matches a broader (and less precise) set of queries, and the LLM has to sift through more irrelevant text once retrieved.
Overlap helps prevent boundary loss but isn't free — 20% overlap means 20% more vectors to store and search, so don't default to it blindly; test with and without on your eval set.

The only reliable way to pick these numbers is to build a small retrieval eval set — twenty to fifty realistic queries with known correct source passages — and measure retrieval hit rate at different chunk sizes. Anecdotally, most production systems converge somewhere in the 300-800 token range for chunk size, but "most systems" is not your system, and the only evaluation that matters is the one run on your documents with your actual query patterns.

Metadata and chunk context are part of chunking too

A mistake I see constantly: teams treat chunking purely as a text-splitting problem and forget to attach metadata that makes each chunk useful after retrieval. A chunk that says "the fee is waived in this case" is useless without knowing *which document*, *which section*, and *which version* it came from. At minimum, every chunk should carry:

Source document name and, ideally, a stable document ID
Section or heading path (e.g. Billing > Refunds > Enterprise Plans)
Position metadata (chunk index, page number if applicable)
Last-updated timestamp, especially for policy or pricing documents that change

chunk_record = {
    "text": chunk_text,
    "metadata": {
        "source": "refund_policy_v3.pdf",
        "section": "Billing > Refunds > Enterprise Plans",
        "chunk_index": 12,
        "updated_at": "2026-05-14",
    },
}

This metadata does double duty. It lets you filter retrieval by recency or section before the LLM even sees the text, and it gives the LLM enough grounding to cite its source accurately in the generated answer — which matters a lot once you're answering questions about anything with an "as of" date, like pricing or compliance policy.

It's also worth deciding early whether metadata lives alongside the chunk in your vector store's payload or in a separate relational table keyed by chunk ID. Embedding it directly in the vector store payload is simpler to query at retrieval time, but if you need to bulk-update metadata — say, correcting a section name across thousands of chunks after a docs reorg — a separate table with a foreign key back to the chunk is much easier to maintain than rewriting vectors you don't actually need to touch.

Common mistakes that undermine any chunking strategy

A few failure patterns show up regardless of which strategy you pick, and they're worth calling out because they're easy to miss in a demo but expensive in production:

Chunking before cleaning. Running a splitter over raw HTML or PDF-extracted text full of navigation boilerplate, repeated headers, and broken whitespace bakes noise into every chunk. Clean and normalize text first.
Ignoring tables and code blocks. A recursive splitter with naive separators will happily cut a markdown table in half. Detect structured blocks and either keep them as atomic chunks or chunk them with table-aware logic.
One chunk size for every document type. A pricing FAQ and a 40-page compliance PDF do not want the same chunk_size. Segment your ingestion pipeline by document type.
Never re-evaluating after a chunking change. Teams tune chunk size once at launch and never revisit it, even as document types in the corpus shift over time. Treat your retrieval eval set as a living asset, not a one-time gate.
Skipping overlap entirely to save storage. This is the single easiest way to silently lose the sentence sitting right at a chunk boundary — and it's often the sentence with the actual answer in it.

Closing thoughts

Chunking doesn't get the attention it deserves because it's not glamorous — there's no leaderboard for it the way there is for embedding models or LLMs. But in almost every RAG system I've debugged, the fix that moved the needle most wasn't a better model or a fancier reranker, it was going back to the chunking step and asking whether the units being retrieved were actually coherent, self-contained, and sized right for the query patterns in play. Start with recursive chunking as your default, reach for semantic chunking when your documents genuinely drift in topic, and save agentic chunking for the high-value, slow-changing corpora where retrieval precision is worth the extra spend. If you want to go deeper on how these chunking decisions fit into the rest of the retrieval pipeline — embeddings, vector stores, reranking, and evaluation — that's exactly the ground we cover hands-on in Introduction to RAG, where you'll build and evaluate each of these chunking strategies against real document sets rather than toy examples.