RAG Cross Encoder Reranking

RAG cross encoder reranking adds a second, more accurate relevance model after fast vector retrieval, so the generator receives fewer weak passages and more evidence that actually answers the query. The practical pattern is retrieve many candidates with a bi-encoder, score each query-document pair with a cross encoder, then send only the best few passages to the LLM. This guide builds that pipeline, exposes its production tradeoffs, and shows how to evaluate it without relying on invented benchmark claims.

Why rag cross encoder reranking improves retrieval

Vector search normally embeds a query and every document independently. Because document vectors can be computed once and indexed, approximate nearest-neighbor search remains fast across millions of chunks. The limitation is architectural: the query and document do not interact token by token during scoring. A passage can be semantically nearby while still missing the exact constraint, entity, time period, or negation in the question.

A cross encoder receives the query and candidate passage together. Its transformer attention can compare words and phrases across both inputs before producing one relevance score. That deeper interaction is usually too expensive for first-stage retrieval, but it is practical for reranking tens of candidates.

The two stages therefore have different jobs:

• The retriever maximizes recall. It should cheaply find a candidate set likely to contain the answer.
• The reranker improves precision. It sorts that small set using richer query-document interaction.
• The generator synthesizes an answer only from the highest-ranked evidence.

This separation matters. Reranking cannot recover a relevant chunk that the first stage never retrieved. Conversely, retrieving 100 candidates does not help the LLM if irrelevant text crowds the context window. A strong pipeline retrieves broadly, reranks carefully, and keeps the final context deliberately small.

Build a minimal reranking environment

Use Python 3.11 or newer and an isolated environment. The example uses Sentence Transformers for embedding and cross-encoder inference, FAISS for local vector search, and NumPy for array handling. Pin versions in a real service after testing them in your environment.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install sentence-transformers faiss-cpu numpy

Create rerank_demo.py. The corpus is intentionally small, but the control flow is the same when FAISS is replaced by a managed vector database.

from dataclasses import dataclass
import faiss
import numpy as np
from sentence_transformers import CrossEncoder, SentenceTransformer


@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    text: str


chunks = [
    Chunk("auth-1", "Access tokens expire after 15 minutes by default."),
    Chunk("auth-2", "Refresh tokens can obtain a new access token without login."),
    Chunk("auth-3", "Administrators can revoke refresh tokens from the security console."),
    Chunk("api-1", "API keys authenticate server-to-server requests and do not expire automatically."),
    Chunk("ui-1", "A user session ends after 30 minutes of browser inactivity."),
    Chunk("audit-1", "Token creation and revocation events are written to the audit log."),
]

embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

documents = [chunk.text for chunk in chunks]
document_vectors = embedder.encode(
    documents,
    normalize_embeddings=True,
    convert_to_numpy=True,
).astype("float32")

index = faiss.IndexFlatIP(document_vectors.shape[1])
index.add(document_vectors)


def retrieve(query: str, candidate_k: int = 5) -> list[dict]:
    query_vector = embedder.encode(
        [query],
        normalize_embeddings=True,
        convert_to_numpy=True,
    ).astype("float32")
    scores, positions = index.search(query_vector, candidate_k)
    return [
        {
            "chunk": chunks[int(position)],
            "retrieval_score": float(score),
        }
        for score, position in zip(scores[0], positions[0])
        if position >= 0
    ]


def rerank(query: str, candidates: list[dict], final_k: int = 3) -> list[dict]:
    pairs = [(query, item["chunk"].text) for item in candidates]
    scores = reranker.predict(pairs)
    ranked = []
    for item, score in zip(candidates, scores):
        ranked.append({**item, "rerank_score": float(score)})
    ranked.sort(key=lambda item: item["rerank_score"], reverse=True)
    return ranked[:final_k]


query = "How can a service get a new access token without asking the user to log in?"
candidates = retrieve(query, candidate_k=5)
results = rerank(query, candidates, final_k=3)

for rank, result in enumerate(results, start=1):
    print(
        rank,
        result["chunk"].chunk_id,
        round(result["retrieval_score"], 4),
        round(result["rerank_score"], 4),
        result["chunk"].text,
    )

Run it with:

python rerank_demo.py

Do not compare the numerical values of the retrieval score and reranker score. They come from different models and have different scales. Use each score for ordering within its own stage unless you have calibrated a combination on labeled data.

Connect reranked passages to a generator

Treat context construction as an explicit function, not an incidental string join. Include stable chunk identifiers so the model can cite evidence and your logs can reconstruct each response. Put a hard character or token budget on context assembly because final_k alone does not control length.

def build_context(results: list[dict], max_chars: int = 4000) -> str:
    blocks = []
    used = 0
    for item in results:
        chunk = item["chunk"]
        block = f"SOURCE {chunk.chunk_id}\n{chunk.text}\n"
        if used + len(block) > max_chars:
            break
        blocks.append(block)
        used += len(block)
    return "\n".join(blocks)


def build_messages(query: str, context: str) -> list[dict]:
    return [
        {
            "role": "system",
            "content": (
                "Answer only from the supplied sources. "
                "If the sources are insufficient, say what is missing. "
                "Cite source IDs for factual claims."
            ),
        },
        {
            "role": "user",
            "content": f"QUESTION\n{query}\n\nSOURCES\n{context}",
        },
    ]


context = build_context(results)
messages = build_messages(query, context)

Pass messages to the supported chat or responses interface for your chosen model provider. Keep provider calls outside retrieval code. That boundary makes it easy to test ranking without spending generation tokens and to change the generator without rebuilding the index.

The prompt should not ask the model to rescue poor retrieval. If evidence is insufficient, return an abstention or launch a controlled fallback, such as a larger candidate pool, keyword retrieval, query rewriting, or a human-review path.

Tune rag cross encoder reranking parameters

Three values dominate quality, latency, and cost: candidate_k, final_k, and chunk size.

candidate_k determines how much work reaches the cross encoder. Start with a range such as 10, 20, 50, and 100, then evaluate on your own queries. Raising it helps only while relevant evidence is being missed by the smaller pool. Cross-encoder work grows roughly with the number and token length of candidate pairs, so large values can quickly exhaust latency budgets.

final_k controls what the generator sees. More context is not automatically better. Extra passages can introduce contradictions, stale policies, and nearby but irrelevant terminology. Tune final_k against answer quality and groundedness, not retrieval metrics alone.

Chunk size affects both stages. Tiny chunks lack enough context to judge relevance. Huge chunks dilute the relevant sentence and consume reranker tokens. A useful implementation stores a compact searchable chunk plus metadata that can expand to a neighboring paragraph after ranking. This preserves scoring precision while giving the generator readable context.

Also consider these controls:

• Deduplicate near-identical chunks before reranking so repeated boilerplate cannot occupy every top position.
• Apply access-control filters before retrieval or immediately after it, never after generation.
• Keep document titles or section paths with the chunk when they carry useful meaning.
• Truncate pairs consciously. Cross encoders have input limits, and silent tokenizer truncation may remove the answer-bearing text.
• Batch pairs to improve accelerator utilization while respecting memory limits.
• Cache scores for repeated query-document pairs only when document versions and permissions are part of the cache key.

Add hybrid retrieval before reranking

Dense embeddings are strong at semantic similarity, while lexical retrieval remains valuable for error codes, product names, identifiers, and exact phrases. A hybrid candidate set gives the cross encoder evidence from both paths.

One simple fusion method is reciprocal rank fusion. It combines ranks rather than incompatible raw scores.

from collections import defaultdict


def reciprocal_rank_fusion(
    ranked_lists: list[list[str]],
    rank_constant: int = 60,
) -> list[tuple[str, float]]:
    fused = defaultdict(float)
    for ranked_ids in ranked_lists:
        for rank, chunk_id in enumerate(ranked_ids, start=1):
            fused[chunk_id] += 1.0 / (rank_constant + rank)
    return sorted(fused.items(), key=lambda item: item[1], reverse=True)


dense_ids = ["auth-2", "auth-1", "ui-1"]
keyword_ids = ["auth-2", "audit-1", "auth-3"]
print(reciprocal_rank_fusion([dense_ids, keyword_ids]))

In production, retrieve independently from the dense and lexical indexes, fuse their identifiers, fetch the corresponding text, deduplicate, then cross-encode the merged pool. Keep the source rank and retrieval channel in tracing data. That makes failures diagnosable, for example, an exact identifier found lexically but pushed down by reranking.

Metadata filtering should happen as early as the retrieval system allows. Tenant, language, region, product version, and document status are often hard constraints, not relevance signals. A reranker should not decide whether an employee may see a confidential passage or whether an obsolete manual is eligible.

Batch cross-encoder inference for throughput

Interactive traffic benefits from small dynamic batches, while offline indexing evaluation can use larger batches. Sentence Transformers exposes batch_size on prediction. Measure it on the hardware and sequence lengths you will actually serve.

def rerank_batched(
    query: str,
    candidates: list[dict],
    final_k: int = 5,
    batch_size: int = 32,
) -> list[dict]:
    pairs = [(query, item["chunk"].text) for item in candidates]
    scores = reranker.predict(
        pairs,
        batch_size=batch_size,
        show_progress_bar=False,
    )
    order = np.argsort(-np.asarray(scores))[:final_k]
    return [
        {
            **candidates[int(i)],
            "rerank_score": float(scores[int(i)]),
        }
        for i in order
    ]

For a service, load the model once at process startup. Do not instantiate it per request. Warm it with representative sequence lengths before accepting traffic, enforce queue limits, and return a documented fallback when the reranker is unavailable. Depending on the application, the fallback can use first-stage ranking or fail closed.

GPU inference is not automatically faster for tiny candidate sets because transfer and scheduling overhead can dominate. CPU inference can be attractive for predictable low-volume workloads. Test p50, p95, and p99 latency under realistic concurrency, not from a single warm notebook request.

Evaluate retrieval and answer quality

Build an evaluation set from real user questions, including typos, underspecified requests, negative questions, and version-specific terms. For each query, label every passage that contains sufficient supporting evidence. If relevance is graded, distinguish a directly answering passage from a merely related one.

Measure both pipeline stages:

• Candidate recall at candidate_k asks whether first-stage retrieval found at least one relevant passage.
• Mean reciprocal rank rewards placing the first relevant result near the top.
• Normalized discounted cumulative gain supports graded relevance across several positions.
• Precision at final_k checks how much of the delivered context is relevant.
• Answer correctness and citation support test the complete RAG system.
• Abstention quality tests whether the system refuses when evidence is absent.

The following small evaluator calculates reciprocal rank and hit rate without another dependency.

def reciprocal_rank(ranked_ids: list[str], relevant_ids: set[str]) -> float:
    for rank, chunk_id in enumerate(ranked_ids, start=1):
        if chunk_id in relevant_ids:
            return 1.0 / rank
    return 0.0


def evaluate(cases: list[dict]) -> dict[str, float]:
    rr_values = []
    hits = []
    for case in cases:
        candidates = retrieve(case["query"], candidate_k=5)
        ranked = rerank(case["query"], candidates, final_k=3)
        ranked_ids = [item["chunk"].chunk_id for item in ranked]
        relevant = set(case["relevant_ids"])
        rr_values.append(reciprocal_rank(ranked_ids, relevant))
        hits.append(float(any(item in relevant for item in ranked_ids)))
    return {
        "mrr": sum(rr_values) / len(rr_values),
        "hit_rate": sum(hits) / len(hits),
    }


cases = [
    {
        "query": "Get a new token without another login",
        "relevant_ids": ["auth-2"],
    },
    {
        "query": "Where are revocation events recorded?",
        "relevant_ids": ["audit-1"],
    },
]
print(evaluate(cases))

Compare at least three configurations: first-stage retrieval alone, retrieval plus reranking, and your proposed fallback path. Freeze the evaluation set before tuning a release. Record model identifiers, tokenizer versions, chunking configuration, index snapshot, and code revision so results are reproducible.

Avoid evaluating only questions whose answer text closely matches the query. Cross encoders often earn their keep on harder cases involving paraphrases, competing passages, and subtle constraints. Also maintain a small adversarial set for prompt injection in retrieved text, stale versions, duplicate chunks, and documents the requesting principal cannot access.

Operate the reranker in production

Instrument each stage separately. A single end-to-end duration hides whether time was spent embedding, searching, fetching documents, reranking, or generating. For every request, capture safe diagnostic fields such as candidate count, pair token lengths, selected chunk IDs, model version, stage durations, and fallback status. Do not log sensitive query or document text unless policy explicitly permits it.

Watch distributions, not only averages. A few oversized chunks can create severe tail latency. Alerts should cover queue depth, timeout rate, empty candidate rate, reranker error rate, and the share of traffic using fallback behavior.

Version the reranker as part of the retrieval stack. A model change can reorder context even when the generator remains unchanged. Roll out with shadow scoring or a limited traffic slice, compare ranking and answer outcomes, then promote deliberately. Keep the previous model and configuration available for rollback.

Security remains a pipeline concern. Retrieved documents are untrusted input, even when they came from an internal index. Separate instructions from sources in the prompt, tell the generator that source text cannot override system instructions, sanitize dangerous rendering, and preserve authorization checks independently of model scores.

Common failure modes and fixes

The best passage never appears after reranking. Inspect first-stage candidate recall. Increase candidate_k, add lexical retrieval, improve chunking, or fix metadata filters. Changing the reranker cannot score a missing passage.

Every candidate receives a similar score. Confirm that the correct query and full passage are paired, inspect truncation, and test queries with clear positive and negative examples. The model may also be mismatched to the language or domain.

Latency grows unpredictably. Log pair token counts, cap chunk length, batch inference, set queue deadlines, and isolate model loading from request handling. Candidate count alone does not represent compute when passage lengths vary.

Reranking improves retrieval metrics but answers get worse. Examine final context diversity, ordering, contradictory versions, and prompt behavior. The top few passages may all repeat one fact while excluding complementary evidence needed for a complete answer.

Offline gains disappear online. Check query distribution drift, permissions, index freshness, and logging differences. Ensure the online service uses the same tokenizer, model revision, and preprocessing evaluated offline.

FAQ

What is a cross encoder in RAG?

A cross encoder jointly processes a query and one candidate passage to produce a relevance score. It is more computationally intensive than independent embeddings, so it usually reranks a small candidate set rather than searching the entire corpus.

How many documents should I rerank?

There is no universal number. Choose candidate_k by measuring candidate recall and latency on representative queries. Increase it until additional candidates stop producing useful improvements within the service budget.

Should reranker scores be used as confidence scores?

Not by default. Raw scores are model-specific ranking signals and may not be calibrated probabilities. If a threshold controls abstention or another business decision, calibrate and validate it on held-out domain data.

Can a cross encoder replace vector search?

Usually not at corpus scale. Cross encoding requires a forward pass for every query-document pair. Vector or lexical retrieval efficiently narrows the corpus, after which cross encoding becomes affordable.

Does reranking prevent hallucinations?

No. It can improve the evidence supplied to the generator, but grounded prompting, citation checks, abstention behavior, authorization, and answer evaluation remain necessary.

When should I skip reranking?

Skip it when first-stage ranking already meets measured quality targets, the latency budget is extremely tight, candidate passages are trivial to distinguish, or the deployment cannot operate the additional model reliably. The correct decision comes from an ablation on real traffic patterns.

How do I choose a reranker model?

Shortlist models that support your language, license requirements, sequence length, deployment hardware, and domain. Then compare them on the same frozen relevance set, including throughput and tail latency. Model popularity is not a substitute for workload-specific evaluation.

Where should reranking happen in the request flow?

Apply hard authorization and eligibility filters first, perform dense and optional lexical retrieval, merge and deduplicate candidates, rerank the remaining passages, enforce the final context budget, and only then call the generator.