RAG Reranking Compared: Cohere, BGE, and Jina

RAG reranking is the second-stage retrieval step that reorders the documents your vector search pulled back, using a model that reads the query and each candidate together instead of comparing precompressed vectors. It fixes the single biggest weakness of embedding-only retrieval: cosine similarity is a rough proxy for relevance, and it regularly ranks a document that shares vocabulary with the query above a document that actually answers it. This article compares three of the most common rerankers engineers reach for, Cohere Rerank, BGE reranker, and Jina Reranker, and shows how to wire each into a retrieval pipeline, what they cost in latency, and when to pick one over another.

Why RAG Reranking Exists

A typical RAG pipeline does retrieval in two passes for a reason. The first pass, usually a vector search over an index like pgvector, Qdrant, or Pinecone, has to be fast across millions of chunks, so it uses bi-encoders: the query and every document are embedded separately into fixed-size vectors, and similarity is just a dot product. That's cheap, but it throws away all the interaction between query terms and document terms. Two passages can land near each other in embedding space because they're topically similar, even if one doesn't answer the question at all.

A reranker is a cross-encoder. It takes the query and a candidate passage together as one input, runs them jointly through a transformer, and outputs a single relevance score. That joint attention is what makes cross-encoders so much more accurate than bi-encoders at judging "does this passage answer this question" rather than "is this passage about the same topic." The cost is that a cross-encoder can't be precomputed and indexed the way embeddings can. You have to run it at query time, once per candidate, which is why reranking only makes sense on a shortlist, not the whole corpus.

The standard pattern:

1. Vector search retrieves top-k candidates (k = 50 to 200) using cheap bi-encoder similarity.
2. A reranker scores each candidate against the query with a cross-encoder.
3. You keep the top-n (n = 3 to 10) reranked results and pass those to the LLM as context.

This is where RAG reranking earns its keep: it's the step that decides what the model actually sees, and a bad top-n selection means a confidently wrong answer no matter how good your prompt is.

How Reranking Fits Into a RAG Pipeline

Before comparing vendors, it helps to be precise about where reranking sits and what it does not do:

• It does not replace the vector index. You still need fast approximate nearest neighbor search to get from millions of chunks down to a manageable candidate set.
• It does not fix bad chunking. If your chunks are too large, too small, or split mid-thought, no reranker can recover information that got cut in half.
• It does not eliminate the need for good embeddings. A reranker can only reorder what the first stage retrieved. If the true answer never makes it into the top-k candidate set, reranking cannot rescue it, so k needs to be generous enough to include the right passage most of the time.
• It adds latency. Every candidate you rerank is an extra forward pass through a transformer, and that adds up when k is large.

With that scoped, here's what each of the three tools actually does.

Cohere Rerank

Cohere Rerank is a hosted, API-only reranking model. You send a query and a list of documents, get back scores and a reordered list. There's no model to download or host, so it's the fastest path to adding reranking to an existing pipeline.

pip install cohere

import cohere

co = cohere.Client("YOUR_COHERE_API_KEY")

query = "How do I rotate a JWT signing key without downtime?"
documents = [
    "JWT tokens can be revoked by maintaining a denylist in Redis.",
    "To rotate signing keys with zero downtime, publish the new public "
    "key alongside the old one in your JWKS endpoint, start signing new "
    "tokens with the new key, and only retire the old key after every "
    "issued token using it has expired.",
    "OAuth2 access tokens typically expire after one hour.",
]

results = co.rerank(
    model="rerank-v3.5",
    query=query,
    documents=documents,
    top_n=2,
)

for r in results.results:
    print(r.index, r.relevance_score, documents[r.index][:60])

The API also accepts structured documents (dicts with multiple fields, not just raw text), which is useful when your chunks carry metadata like a title or section header you want the model to consider alongside the body text. Cohere Rerank supports long documents natively; it chunks internally if a document exceeds the model's context window and returns a single score, so you don't have to pre-chunk documents that are already reasonably sized.

Strengths: no infrastructure to manage, consistently strong relevance judgments across domains, multilingual support out of the box, simple integration that's a few lines of code in most RAG frameworks (LangChain and LlamaIndex both ship a CohereRerank wrapper).

Tradeoffs: it's a paid API call on every query, so cost scales with query volume and candidate count. It also means every document you rerank leaves your infrastructure, which matters if you're in a regulated environment or handling sensitive data you can't send to a third party.

BGE Reranker

BGE (BAAI General Embedding) reranker is an open-source cross-encoder family from the Beijing Academy of Artificial Intelligence, distributed as ordinary Hugging Face model weights. You run it yourself, on your own GPU or CPU, with no per-query cost beyond compute.

pip install FlagEmbedding

from FlagEmbedding import FlagReranker

reranker = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)

query = "How do I rotate a JWT signing key without downtime?"
documents = [
    "JWT tokens can be revoked by maintaining a denylist in Redis.",
    "To rotate signing keys with zero downtime, publish the new public "
    "key alongside the old one in your JWKS endpoint, start signing new "
    "tokens with the new key, and only retire the old key after every "
    "issued token using it has expired.",
    "OAuth2 access tokens typically expire after one hour.",
]

pairs = [[query, doc] for doc in documents]
scores = reranker.compute_score(pairs, normalize=True)

ranked = sorted(zip(scores, documents), reverse=True)
for score, doc in ranked:
    print(round(score, 4), doc[:60])

bge-reranker-v2-m3 is multilingual and handles longer passages reasonably well; the BGE family also ships smaller variants (bge-reranker-base, bge-reranker-large) if you need lower latency at some accuracy cost, and a layerwise variant (bge-reranker-v2-minicpm-layerwise) that lets you trade depth for speed at inference time by choosing how many layers to run.

Strengths: no per-query fees, data never leaves your infrastructure, you can fine-tune it on your own query-document pairs if your domain has vocabulary generic rerankers don't handle well (internal tooling docs, legal contracts, medical notes), and you fully control batching and hardware placement.

Tradeoffs: you own the ops. That means provisioning a GPU (CPU inference works but is materially slower), managing model updates, and handling the batching and queueing logic that a hosted API gives you for free. Cold-start latency on a serverless GPU endpoint can be worse than a Cohere API call unless you keep an instance warm.

Jina Reranker

Jina AI's rerankers occupy a middle ground: the model weights are open source and can be self-hosted the same way as BGE, but Jina also offers a hosted Reranker API if you'd rather not manage infrastructure.

Hosted API:

import requests

url = "https://api.jina.ai/v1/rerank"
headers = {
    "Authorization": "Bearer YOUR_JINA_API_KEY",
    "Content-Type": "application/json",
}
payload = {
    "model": "jina-reranker-v2-base-multilingual",
    "query": "How do I rotate a JWT signing key without downtime?",
    "documents": [
        "JWT tokens can be revoked by maintaining a denylist in Redis.",
        "To rotate signing keys with zero downtime, publish the new "
        "public key alongside the old one in your JWKS endpoint.",
        "OAuth2 access tokens typically expire after one hour.",
    ],
    "top_n": 2,
}

response = requests.post(url, headers=headers, json=payload)
for r in response.json()["results"]:
    print(r["index"], r["relevance_score"])

Self-hosted, via sentence-transformers' CrossEncoder interface:

from sentence_transformers import CrossEncoder

model = CrossEncoder(
    "jinaai/jina-reranker-v2-base-multilingual",
    trust_remote_code=True,
)

query = "How do I rotate a JWT signing key without downtime?"
documents = [
    "JWT tokens can be revoked by maintaining a denylist in Redis.",
    "To rotate signing keys with zero downtime, publish the new "
    "public key alongside the old one in your JWKS endpoint.",
    "OAuth2 access tokens typically expire after one hour.",
]

scores = model.predict([[query, doc] for doc in documents])
ranked = sorted(zip(scores, documents), reverse=True, key=lambda x: x[0])
for score, doc in ranked:
    print(round(float(score), 4), doc[:60])

Jina's rerankers are notable for long-context support; several variants are trained to handle much longer input sequences than typical cross-encoders, which matters if your chunks run long or you're reranking whole sections instead of small passages. Jina also publishes multilingual and code-focused variants aimed at reranking source code and technical documentation, which BGE and Cohere handle less specifically.

Strengths: choice of hosted or self-hosted with the same weights, strong long-context handling, specialized variants for code search, competitive multilingual coverage.

Tradeoffs: the hosted API is a smaller, less battle-tested service than Cohere's if you care about production SLAs, and self-hosting carries the same operational burden as BGE.

Latency, Cost, and Accuracy Tradeoffs

These three tools sit on a spectrum, and the right axis to reason about is build-vs-buy, not "which is the best model," because the answer to that shifts every few months as new checkpoints ship.

Cost. Cohere and hosted Jina bill per query or per document processed, so cost is directly proportional to traffic and to how many candidates you rerank per query. BGE and self-hosted Jina shift cost to fixed infrastructure: a GPU instance that's either idle capacity you're paying for anyway, or a new line item you have to justify. At low query volume, hosted APIs are almost always cheaper than provisioning a dedicated GPU. At high, steady query volume, self-hosting usually wins because a GPU that's kept busy amortizes better than per-call pricing.

Latency. Hosted APIs add network round-trip time on top of inference, typically tens to low hundreds of milliseconds depending on candidate count and your region relative to the provider's. Self-hosted rerankers on a warm GPU can be faster in raw inference time, especially with a smaller model variant, but only if you've solved the cold-start and batching problem; a serverless GPU that spins up on demand can be slower than any API call. If your product has a hard latency budget, say sub-second end to end for the full RAG pipeline, this is usually the deciding factor over raw accuracy differences.

Accuracy. All three model families are trained on large-scale relevance data and perform well on standard retrieval benchmarks; the practical accuracy gap between them on general-domain English text is usually smaller than the gap between using a reranker at all versus skipping it. Where the gap does widen is on specialized domains: a BGE reranker fine-tuned on your own labeled query-document pairs will typically outperform any general-purpose hosted model on your specific corpus, because it has actually seen your vocabulary and your notion of relevance. If you can't fine-tune, the choice between Cohere and Jina on accuracy alone is close enough that you should benchmark on your own evaluation set rather than trust a generic leaderboard number.

A Minimal Benchmark Harness

Rather than trusting any vendor's marketing numbers, run your own comparison on a slice of your real queries and known-good documents. Here's a skeleton that scores all three rerankers against the same query set and computes a simple ranking metric.

import time

def evaluate_reranker(rerank_fn, eval_set):
    """eval_set: list of (query, documents, relevant_doc_index)"""
    hits_at_1 = 0
    hits_at_3 = 0
    total_latency = 0.0

    for query, documents, relevant_index in eval_set:
        start = time.perf_counter()
        ranked_indices = rerank_fn(query, documents)
        total_latency += time.perf_counter() - start

        if ranked_indices[0] == relevant_index:
            hits_at_1 += 1
        if relevant_index in ranked_indices[:3]:
            hits_at_3 += 1

    n = len(eval_set)
    return {
        "hits_at_1": hits_at_1 / n,
        "hits_at_3": hits_at_3 / n,
        "avg_latency_ms": (total_latency / n) * 1000,
    }

Wrap each of the three integrations shown above in a function with the signature rerank_fn(query, documents) -> list[int] (a list of document indices sorted by descending relevance), build an eval set of maybe 50 to 100 real queries paired with the chunk you know contains the answer, and run all three through this harness. That gives you numbers grounded in your own data instead of a benchmark that may not reflect your domain.

Choosing Between Cohere, BGE, and Jina

A few concrete rules of thumb, based on the tradeoffs above:

• Prototyping or low query volume: use Cohere Rerank. It's the least code, no infrastructure, and the API's default model is a reasonable choice on general-domain text.
• High, steady query volume where cost per query matters: self-host BGE reranker. Once you're pushing enough traffic that GPU utilization stays healthy, the per-query cost of a hosted API adds up faster than owning the box.
• Data residency or compliance constraints: self-host BGE or Jina. Neither sends your document text to a third party.
• Long chunks or code search: lean toward Jina's reranker variants, which are trained with longer context windows and have code-specific checkpoints that general rerankers don't offer.
• A narrow, specialized domain with labeled data available: fine-tune BGE reranker on your own query-document pairs. This is the one scenario where a smaller, tuned open model reliably beats any general-purpose hosted API.
• You don't know yet: start with Cohere for speed of integration, instrument the pipeline to log query/candidate/chosen-context triples, and revisit the decision once you have real traffic and a real eval set. Swapping a reranker later is a small, isolated change if you kept the interface generic.

Common Pitfalls in RAG Reranking

Reranking too few candidates. If your vector search only returns k=5 and the right passage is ranked 8th by the bi-encoder, no reranker will ever see it. Retrieve generously (k=50 to 100) before reranking down to n=3 to 10.

Reranking too many candidates. Every candidate is a forward pass. Reranking 500 documents per query on a hosted API is both slow and expensive; tune k to the smallest value that reliably contains the right answer in your eval set.

Ignoring the score, not just the order. Most rerankers return a relevance score, not just a rank. Use it: if the top result's score is low in absolute terms, that's a signal your retrieval failed entirely, and it's often better to say "I don't have enough context" than to hand the LLM a low-relevance passage and let it hallucinate an answer anyway.

Reranking on stale chunks. If your reranker was fine-tuned or evaluated against one chunking strategy and you later change chunk size or overlap, re-run your eval set. Reranker performance is sensitive to how much context is packed into each candidate.

Treating reranking as a fix for bad retrieval. If your bi-encoder embeddings are a poor fit for your domain (common with highly technical or jargon-heavy corpora), a reranker will improve the final order but can't compensate for a candidate set that's missing the right document altogether. Fix retrieval quality first, then layer reranking on top.

FAQ

Does RAG reranking always improve answer quality? In most retrieval-heavy pipelines, yes, because it directly improves what context reaches the LLM. But it adds latency and, for hosted rerankers, cost, so measure the improvement on your own eval set rather than assuming it. If your first-stage retrieval already returns the right passage in the top 2 or 3 results most of the time, reranking has less room to help.

How many candidates should I send to a reranker? A common starting point is k=50 to 100 candidates in, top n=3 to 10 out, then adjust based on your eval set and latency budget. Larger k improves recall at the reranking stage but costs more compute per query.

Can I use a reranker without an embedding-based vector search at all? Yes, technically you could rerank the output of keyword search (BM25) or any other candidate-generation method. The two-stage pattern, cheap broad retrieval followed by expensive precise reranking, applies regardless of what the first stage is.

Is BGE reranker as good as Cohere Rerank? On general benchmarks the two are close, and the practical difference on your own data is often smaller than the difference between using a reranker versus not using one. The honest answer is to benchmark both on your own query set; the gap that matters most shows up in specialized domains, where a fine-tuned BGE model tends to pull ahead of any general-purpose hosted model.

Do I need a GPU to self-host BGE or Jina rerankers? No, CPU inference works, especially with the smaller model variants, but latency will be noticeably higher than GPU inference, particularly as candidate count grows. For anything beyond low-traffic or offline batch use, a GPU is worth it.

Should reranking run synchronously in the request path, or can it be cached? It typically runs synchronously since it depends on the live query, but you can cache reranked results for repeated or templated queries, and you can parallelize reranking across candidates if your reranker and infrastructure support batched inference, which most do.