RAGembeddingschunkingvector searchretrieval

Late Chunking and Long-Context Embeddings Explained

Pramod Dutta · Jul 8, 2026 · 18 min read

Late chunking reverses the standard RAG indexing pipeline: instead of splitting a document into chunks and embedding each chunk in isolation, you run the entire document through a long-context embedding model first, take the contextual token embeddings that come out of the transformer, and only then cut them into chunk spans and mean-pool each span into one vector. You still end up with one embedding per chunk, stored in the same vector database and queried the same way. The difference is that every chunk vector was computed while the model could attend to the whole document, so a sentence like "The city has around 3.85 million inhabitants" lands close to the query "Berlin population" even though the word Berlin never appears in that sentence.

The technique came out of Jina AI in 2024, and it has since moved from paper trick to first-class API feature and a standard option in serious RAG stacks. This article covers how late chunking works mechanically, a runnable implementation on open weights, an honest comparison against contextual retrieval and ColBERT-style late interaction, which models support it, what to do with documents longer than the context window, and how to verify the gain on your own corpus instead of trusting benchmarks.

Why chunk-then-embed loses information

The default RAG pipeline chunks first and embeds second. You split a document into 200 to 500 token pieces, feed each piece to the embedding model separately, and store the vectors. The model never sees a single token outside the chunk it is currently embedding.

That is a real information loss, and it shows up in predictable places:

Pronouns and definite references. "It", "she", "the company", "the city", "this approach". The entity lives in an earlier chunk; the chunk you actually need at query time only contains the reference.
Defined terms. A contract defines "the Supplier" on page 1 and uses it for forty pages. Every chunk after page 1 embeds as if it were about an anonymous supplier.
Abbreviations expanded once. A paper spells out "retrieval-augmented generation (RAG)" in the intro and writes RAG afterwards. Chunks from the middle of the paper never see the expansion.
Section context. "Returns a 402 on failure" means something different under a heading called "Billing API" than under "Parking garage firmware". The heading sits in another chunk.

The classic workarounds all have costs. Bigger chunks reduce the number of broken references but dilute the vector: a 2000 token chunk covering five topics matches precise queries worse than a focused sentence does. Overlapping windows duplicate storage and still cap context at the window size. Prepending the document title or section heading to every chunk helps and is cheap, but it is manual and carries only a few tokens of context.

Late chunking attacks the root cause: the embedding model should see the document, not the fragment.

What late chunking actually does

The trick relies on a property most people ignore: transformer embedding models produce an embedding for every token, and the single vector you get back is just a pooled summary of those token embeddings (for most retrieval models, the mean). Pooling is the last step. Late chunking moves the chunking decision after the transformer and before the pooling:

Tokenize the full document, up to the model's maximum length (8192 tokens for most current long-context embedders, roughly a dozen pages of dense prose).
Run one forward pass. Every token embedding that comes out is conditioned on the entire document through self-attention.
Decide chunk boundaries (sentences, paragraphs, fixed windows: whatever your splitter produces) and map them to token spans.
Mean-pool each token span into one chunk vector.
Store the vectors exactly as you would with naive chunking.

The name is literal: the chunking happens late. Naive chunking pools over tokens that only ever saw their own chunk; late chunking pools over tokens that saw everything. When the model encodes "The city" in step 2, attention has already tied those tokens to "Berlin" from two sentences earlier, so the pooled chunk vector inherits that binding for free.

Two requirements follow directly from the mechanism. You need access to token-level hidden states (an open-weight model, or an API that exposes the feature), and the model should be trained with mean pooling, because pooling arbitrary spans of token embeddings is exactly what mean pooling already does globally.

A runnable late chunking implementation

Here is the whole thing with an open-weight long-context model. jina-embeddings-v2-base-en handles 8192 tokens with mean pooling, which makes it a clean demo model.

pip install torch transformers

Load the model and write a minimal sentence splitter that returns character spans:

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

MODEL_ID = "jinaai/jina-embeddings-v2-base-en"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModel.from_pretrained(MODEL_ID, trust_remote_code=True)
model.eval()

def split_sentences(text):
    # demo splitter: use a real sentence segmenter in production
    spans, start = [], 0
    for i, ch in enumerate(text):
        if ch == "." and (i + 1 == len(text) or text[i + 1] == " "):
            spans.append((start, i + 1))
            start = i + 2
    if start < len(text):
        spans.append((start, len(text)))
    return spans

The late chunking function embeds the full text once, then pools token spans. The tokenizer's offset mapping translates character boundaries into token indices; special tokens like the leading CLS report a zero-width offset and get filtered out:

@torch.no_grad()
def embed_late(text, char_spans):
    enc = tokenizer(
        text,
        return_tensors="pt",
        return_offsets_mapping=True,
        truncation=True,
        max_length=8192,
    )
    offsets = enc.pop("offset_mapping").squeeze(0).tolist()
    token_emb = model(**enc).last_hidden_state.squeeze(0)

    vectors = []
    for c_start, c_end in char_spans:
        idx = [
            i
            for i, (t_start, t_end) in enumerate(offsets)
            if t_start >= c_start and t_end <= c_end and t_end > t_start
        ]
        vectors.append(token_emb[idx].mean(dim=0))
    return torch.stack(vectors)

The naive baseline embeds every chunk separately with the same model and the same pooling, so the only variable is when the chunking happens:

@torch.no_grad()
def embed_naive(text, char_spans):
    vectors = []
    for c_start, c_end in char_spans:
        enc = tokenizer(
            text[c_start:c_end],
            return_tensors="pt",
            truncation=True,
            max_length=8192,
        )
        hidden = model(**enc).last_hidden_state.squeeze(0)
        vectors.append(hidden[1:-1].mean(dim=0))  # drop CLS and SEP
    return torch.stack(vectors)

Now the canonical test: a paragraph where two of the three sentences refer to Berlin only through "Its" and "The city":

DOC = (
    "Berlin is the capital and largest city of Germany, both by area "
    "and by population. Its more than 3.85 million inhabitants make it "
    "the most populous city in the European Union as measured by "
    "population within city limits. The city is also one of the states "
    "of Germany and covers an area of roughly 891 square kilometers."
)
QUERY = "What is the population of Berlin?"

spans = split_sentences(DOC)
chunks = [DOC[s:e] for s, e in spans]

late_vecs = embed_late(DOC, spans)
naive_vecs = embed_naive(DOC, spans)

q_enc = tokenizer(QUERY, return_tensors="pt")
with torch.no_grad():
    q_vec = model(**q_enc).last_hidden_state.squeeze(0)[1:-1].mean(dim=0)

print("naive   late    chunk")
for chunk, nv, lv in zip(chunks, naive_vecs, late_vecs):
    n_sim = F.cosine_similarity(q_vec, nv, dim=0).item()
    l_sim = F.cosine_similarity(q_vec, lv, dim=0).item()
    print(f"{n_sim:.3f}   {l_sim:.3f}   {chunk[:60]}")

Run it and read the two columns. The first sentence scores roughly the same in both pipelines because it names Berlin explicitly. The population sentence and the "The city" sentence score clearly higher under late chunking: their tokens were encoded with Berlin inside the attention window, so the pooled vectors carry the entity even though the surface text does not. That is the entire effect, demonstrated in about sixty lines.

One implementation note: tokens that straddle a chunk boundary get dropped by the span filter above. With sentence or paragraph boundaries this almost never happens; if you chunk mid-word for some reason, assign the straddling token to one side explicitly.

Late chunking with the Jina API

If you do not want to host a model, the Jina embeddings API ships late chunking as a flag. When late_chunking is true, all inputs in one request are treated as consecutive chunks of a single document: the API concatenates them, encodes them in one shared context, then returns one contextualized vector per input.

curl https://api.jina.ai/v1/embeddings \
  -H "Authorization: Bearer $JINA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jina-embeddings-v3",
    "task": "retrieval.passage",
    "late_chunking": true,
    "input": [
      "Berlin is the capital and largest city of Germany.",
      "Its more than 3.85 million inhabitants make it the most populous city in the European Union.",
      "The city is also one of the states of Germany."
    ]
  }'

Two things to keep straight. First, group only chunks of the same document into one request; mixing documents poisons the shared context. Second, the combined inputs must fit the model's context window, so a long document still needs the macro-chunking strategy described below. Embed queries as plain single inputs with task set to retrieval.query and without the flag; queries are short and have no surrounding document.

Voyage AI offers the same idea under a different name: its voyage-context-3 model takes chunks grouped by document and returns contextualized chunk embeddings. If you are already on a managed embedding stack, check for that class of feature before deciding to self-host.

Which models support late chunking

Late chunking needs three things from a model:

A genuinely long context window. 8192 tokens is the practical floor; below that there is not enough shared context to be worth the plumbing.
Token-level output access, either through open weights or an API flag.
Mean pooling as the trained pooling scheme, so that pooling arbitrary spans stays in-distribution.

Models that fit today:

jina-embeddings-v2-base-en and its v2 siblings: 8192 tokens, open weights, the original late chunking testbed.
jina-embeddings-v3: 8192 tokens, multilingual, task adapters, Matryoshka dimensions, with late chunking supported natively in the API and reproducible on the open weights.
nomic-embed-text-v1.5: 8192 tokens, open weights, mean pooling. Remember its required prefixes ("search_document: " for passages, "search_query: " for queries) when you adapt the code above.
gte-modernbert-base: 8192 tokens on a ModernBERT backbone, open weights, fast on current GPUs.

Models that do not fit, and why:

OpenAI text-embedding-3-small and text-embedding-3-large return one pooled vector per input and expose no token states, so do-it-yourself late chunking is impossible. The same applies to most closed embedding APIs that lack an explicit flag.
Decoder-based embedders that use last-token pooling (the Qwen3-Embedding family, for example) break the "pool any span" assumption. Mean-pooling a span of their hidden states is off-label behavior; run your own eval before trusting it.
Long-context rerankers and chat LLMs are not substitutes. You specifically need an embedding model whose token states pool into retrieval-quality vectors.

Documents longer than the context window

Real corpora contain 200 page PDFs. When a document exceeds the model window, use long late chunking, which is macro-chunking with overlap:

Split the document into macro chunks that fit the model, for example around 8000 tokens each with a 512 token overlap, cutting at section boundaries where possible.
Run late chunking inside each macro chunk.
For fine-grained chunks that fall inside an overlap region, keep the vector from the macro chunk where they sit furthest from the edge, so every chunk is encoded with as much surrounding context as possible.

You lose cross-references that span macro chunks (a term defined on page 3 and used on page 150 will not be linked), but every chunk still carries several thousand tokens of neighborhood context instead of a few hundred. In practice that covers the large majority of coreference, which is local.

If your documents have strong structure, cut macro chunks at real boundaries: chapters, top-level headings, message threads. Context sharing across a true topic boundary is not just useless, it can blur vectors. A concatenated FAQ dump is the degenerate case: when adjacent items are genuinely unrelated, late chunking has nothing useful to share, and boundary-aligned macro chunks make it behave like naive chunking, which is the correct behavior there.

Late chunking vs contextual retrieval vs ColBERT

Three techniques attack chunk context loss. They are frequently confused, and they have very different cost profiles.

Late chunking:

Mechanism: encode the whole document once, pool chunk spans afterwards.
Index cost: one long-context embedding pass per document. No LLM calls.
Storage: one vector per chunk, identical to naive chunking.
Query path: unchanged single-vector search.
Constraint: needs a mean-pooling long-context embedder with token access.

Contextual retrieval (the Anthropic recipe):

Mechanism: for each chunk, ask an LLM to write a short situating sentence given the full document, prepend it to the chunk text, then embed with any model you like.
Index cost: one LLM call per chunk. Prompt caching on the shared document makes it affordable, but it is still typically the dominant indexing cost.
Storage: one vector per chunk, plus slightly longer stored text.
Extra win: the prepended text also improves BM25 and other lexical retrieval, because the missing entity names now literally appear in the chunk. Late chunking does nothing for BM25; its added context lives only in the vector.
Constraint: works with any embedding model, including closed APIs.

Late interaction (ColBERT and friends):

Mechanism: skip pooling entirely, store a vector per token, and score with MaxSim at query time.
Index cost: comparable encoding cost, but storage explodes to hundreds of vectors per chunk, and you need an index that supports multi-vector scoring or a compression scheme like PLAID.
Quality: the strongest fine-grained matching of the three, especially for queries that mix exact terms with context.
Constraint: operationally the heaviest by far.

A useful mental model: late chunking is the cheapest upgrade because it only changes when pooling happens; contextual retrieval buys lexical gains too but charges LLM tokens for them; late interaction buys the most precision and charges storage and infrastructure for it. They also compose. Late chunked vectors for recall with a cross-encoder reranker on top is a common production shape, and nothing stops you from combining contextual text rewriting with late chunked embeddings, though the overlap in what they fix means you should measure before paying for both.

Cost, storage, and latency

The operational profile is the main reason late chunking is easy to say yes to:

Storage: identical to naive chunking. One vector per chunk, same dimensionality. Your vector database, your HNSW parameters, and your quantization settings all stay put.
Query latency: identical. The query encodes to one vector and the search is unchanged. This is the decisive difference from late interaction.
Index compute: one 8192 token pass instead of sixteen 512 token passes over the same text. Exact attention grows superlinearly with sequence length, so the single long pass costs somewhat more compute and noticeably more GPU memory at indexing time; batch fewer documents per GPU and you are fine. There are no per-chunk LLM calls, which keeps the indexing bill well below contextual retrieval on large corpora.
Update semantics: this is the real gotcha. With naive chunking, editing one paragraph means re-embedding one chunk. With late chunking, every chunk vector in the document depends on the whole document, so a document edit means re-embedding the document, or at least its macro chunk. Treat the document as the unit of indexing, version it, and re-embed on change. For most corpora documents are small enough that this is a non-issue, but design your ingestion around it from day one.

When late chunking helps and when it will not

Expect clear gains when:

Your documents are prose with heavy coreference: reports, contracts, documentation, wiki pages, support threads, meeting notes.
Your chunks are small. Sentence-level and short-paragraph chunks lose the most context under naive embedding, so they gain the most. Late chunking is what makes very fine-grained chunking viable at all, and published ablations consistently show the relative gain growing as chunks shrink.
Queries name entities that documents mention once and then pronoun away. This is the Berlin pattern, and it is everywhere in real corpora: "the patient", "the defendant", "the service".

Expect little or nothing when:

Chunks are naturally self-contained: FAQ entries, product listings, log lines, code functions with no shared file context worth having.
Documents are short enough to be a single chunk anyway. There is no context to share.
Your retrieval bottleneck is elsewhere. If BM25 already wins on your workload because users query exact part numbers, fix hybrid search first.

And watch for the one failure mode: context dilution. Pooling over tokens that attended to genuinely unrelated material can drag chunk vectors toward the document average. You see this with mega-documents that are really concatenations, such as exported chat dumps or scraped listing pages. The fix is structural macro-chunk boundaries, covered above.

Measuring the gain on your own corpus

Do not adopt late chunking off a blog post, including this one. The eval is cheap:

Sample 100 to 200 real queries, from logs if you have them, otherwise written against known documents.
Label the gold chunk or gold document for each query.
Build two indexes over identical chunks with identical boundaries and the same model: one embedded naively, one embedded with late chunking. Change nothing else.
Measure recall at 5, recall at 10, and MRR on both. Slice by document type; the aggregate can hide a big win on contracts and a wash on FAQs.
If late chunking wins on recall, keep your downstream eval running too. Better candidates usually help end-to-end answer quality, but confirm it moves.

The most common eval mistake is comparing late chunking at sentence granularity against naive chunking at 512 tokens. That confounds chunk size with the embedding method. Hold boundaries constant first; tune size separately once you know which method wins at parity.

Wiring it into a production RAG stack

Downstream of the embedding step, nothing changes. A stored chunk looks the same as it always did:

{
  "id": "doc-42#s7",
  "vector": [0.0132, -0.0871, ...],
  "payload": {
    "doc_id": "doc-42",
    "chunk_index": 7,
    "char_start": 3120,
    "char_end": 3310,
    "text": "The city is also one of the states of Germany...",
    "embed_method": "late_chunking_v1"
  }
}

Practical notes from running this in real pipelines:

Store character offsets and an embedding-method tag. When you A/B late chunking against naive embedding, the tag lets both live in one collection behind a filter.
Keep the chunker deterministic and versioned. Late chunking turns boundaries into a pure pooling decision at embed time, but retrieval still returns chunk text to the LLM, so boundary quality still matters for generation.
Batch by document, not by chunk. The unit of work is one forward pass per document or macro chunk, and your queue and retry logic should reflect that.
Fall back gracefully. If a document exceeds the window and macro chunking fails (pathological single-line files, for instance), embed it naively and tag it, rather than dropping the document.
Queries never use late chunking. Embed them as plain single inputs, with the query-side task prefix or adapter if the model has one.

The bottom line

Late chunking is the rare retrieval upgrade with almost no operational downside: same storage, same query path, no LLM indexing bill, one changed step in the embedding worker. It will not fix lexical search, it will not replace a reranker, and it does nothing for corpora of self-contained snippets. But if your RAG system serves prose documents and your failure log is full of chunks that say "it" while the query says "Berlin", it is one of the highest-leverage changes you can ship in a week, and you can validate it with a two-column similarity printout in sixty lines of Python.

FAQ

Is late chunking the same as late interaction? No. Late interaction (ColBERT) stores one vector per token and scores token-to-token at query time, which multiplies storage and requires special index support. Late chunking stores one vector per chunk, exactly like naive chunking; only the encoding order changed. The names sound alike because both delay a step until after the transformer runs, but the storage and serving implications are completely different.

Do I still need a chunking strategy with late chunking? Yes. Chunk boundaries decide what text the LLM receives after retrieval and what granularity your vectors have. Late chunking removes the context penalty of small chunks; it does not remove the need to choose sensible boundaries. Sentence or short-paragraph boundaries aligned to document structure remain a good default.

Can I do late chunking with OpenAI embeddings? No. text-embedding-3-small and text-embedding-3-large return only a pooled vector per input, and the API exposes no token-level states. Use an open-weight long-context embedder, the Jina API flag, or a contextualized chunk model like voyage-context-3. If you are locked into OpenAI embeddings, contextual retrieval is the equivalent lever available to you.

Does late chunking replace a reranker? No, they stack. Late chunking improves first-stage recall by fixing chunk vectors; a cross-encoder reranker improves precision on the retrieved candidates and sees the full text at scoring time anyway. A strong default stack in 2026 is hybrid lexical plus late chunked dense retrieval, fused, then reranked.

How much accuracy does late chunking add? It depends on how much cross-chunk reference your corpus has and how small your chunks are. Published results and the reproducible Berlin-style experiments consistently show gains on long-document retrieval benchmarks, growing as chunks shrink, and roughly neutral results on self-contained chunks. Run the eval recipe above; with 100 labeled queries you get a trustworthy answer for your own data in an afternoon.

What chunk size should I use with late chunking? Smaller than you would dare with naive embedding. Since each vector keeps document-level context regardless, sentence-level or two to three sentence chunks work well and give the LLM precise citations. Keep macro chunks, the unit actually fed to the model, aligned to real document structure and inside the context window, and let the fine-grained chunks ride within them.