RAG

Embedding Models Compared: OpenAI vs Cohere vs Open-Source in 2026

Ira Menon · Jun 9, 2026 · 16 min read

Why embedding choice quietly decides whether your RAG system works

Every RAG tutorial spends ten paragraphs on chunking strategy and vector database selection, then picks an embedding model in a single throwaway line: "we'll use text-embedding-3-small." That line deserves more scrutiny than almost anything else in the pipeline. Your embedding model is the thing that decides whether "cancellation policy" and "how do I get a refund" land near each other in vector space. Get it wrong and no amount of clever reranking or prompt engineering will fully recover the retrieval quality you lost at the very first step.

By 2026, the embedding landscape has matured into three real camps: OpenAI's hosted API models, Cohere's retrieval-tuned family, and a fast-improving open-source ecosystem you can self-host. Each camp optimizes for a different constraint — convenience, retrieval precision, or control and cost — and the "best" one depends entirely on what you're building, not on a leaderboard score. This article walks through how these models actually differ, where each one wins, and how to test them against your own data instead of trusting a benchmark table.

What an embedding model actually does, briefly

An embedding model takes a chunk of text and maps it to a dense vector — typically somewhere between 384 and 3072 numbers — such that semantically similar text ends up close together in that vector space, usually measured with cosine similarity. That's the entire job. Everything else in RAG (chunking, indexing, reranking, generation) is built on top of the assumption that this mapping is faithful to meaning.

Two properties matter more than raw benchmark scores:

Retrieval fidelity: does the model actually separate "relevant" from "irrelevant" for the specific kind of question your users ask (legal clauses, support tickets, code, medical notes)?
Practical fit: does the vector size, latency, cost, and licensing match your deployment reality — a serverless API route, a GPU box in a private VPC, or a laptop-only prototype?

A model that tops the MTEB leaderboard on classification tasks can still underperform a smaller model on your narrow, jargon-heavy support corpus. This is the single most important thing to internalize before you pick anything.

OpenAI's embedding models: the default for a reason

OpenAI's text-embedding-3-small and text-embedding-3-large are still the most common starting point for teams building RAG in 2026, mostly because they're one API call away and integrate cleanly with the rest of the OpenAI stack most teams already use for generation.

text-embedding-3-small: cheap, fast, and good enough for the majority of general-purpose RAG use cases — FAQ bots, internal wikis, product documentation search. Output dimension is configurable down from its native size, which matters a lot for storage costs at scale.

text-embedding-3-large: meaningfully better on nuanced semantic distinctions (subtle differences in intent, longer documents, cross-lingual queries) at a higher per-token cost and larger vector size. Worth it when retrieval precision directly affects revenue or compliance, not worth it for a weekend side project.

The single most underused feature of the -3 generation is Matryoshka-style dimension truncation — you can request a smaller output dimension (say, 256 instead of 1536) and the model still produces a reasonably useful embedding, because it was trained so that the most important information is front-loaded in the vector. This lets you trade a small amount of accuracy for a large reduction in storage and search latency, which matters once you're indexing millions of chunks.

from openai import OpenAI

client = OpenAI()

def embed_openai(texts, model="text-embedding-3-small", dimensions=512):
    response = client.embeddings.create(
        model=model,
        input=texts,
        dimensions=dimensions,  # truncated Matryoshka embedding
    )
    return [item.embedding for item in response.data]

chunks = [
    "Our refund window is 30 days from the delivery date.",
    "How do I return a product I no longer want?",
]
vectors = embed_openai(chunks)
print(len(vectors), len(vectors[0]))  # 2 512

The tradeoffs are the ones you'd expect from any hosted API: you send your data to a third party, you're rate-limited by someone else's infrastructure, and your embedding costs scale linearly with document volume forever — there's no "buy the GPU once" option. For teams that value shipping speed over infrastructure control, this is rarely a dealbreaker. For teams in regulated industries, it can be a hard no.

Cohere's Embed models: built specifically for retrieval

Where OpenAI's embeddings are general-purpose (they also do reasonably well on clustering and classification), Cohere's embed-v4 family is explicitly optimized around retrieval and reranking workflows — which is exactly what RAG is. This shows up in a few concrete ways rather than just marketing language.

Input type parameters. Cohere's API asks you to declare whether a given text is a search_document (something you're indexing) or a search_query (something a user typed). The model then applies asymmetric encoding tuned for each role. This matters because queries and documents are linguistically different — a query is short and imprecise ("cancel my plan"), a document is long and precise ("Subscription cancellation must be requested in writing..."). Treating them identically, as many models do, leaves retrieval quality on the table.

Multilingual strength. Cohere has invested heavily in non-English retrieval quality, which is a real differentiator if your knowledge base spans multiple languages or your users query in a language different from your source documents.

Compressed embeddings. Cohere supports int8 and binary embedding output natively, which can cut vector storage by 4x to 32x with a modest retrieval quality cost — genuinely useful once your index crosses tens of millions of vectors and storage/search-latency costs start to dominate your infrastructure bill.

import cohere

co = cohere.Client("YOUR_COHERE_API_KEY")

def embed_cohere(texts, input_type="search_document"):
    response = co.embed(
        texts=texts,
        model="embed-v4.0",
        input_type=input_type,
        embedding_types=["float"],
    )
    return response.embeddings.float

docs = ["Subscription cancellation must be requested in writing 30 days prior."]
query = ["cancel my plan"]

doc_vecs = embed_cohere(docs, input_type="search_document")
query_vecs = embed_cohere(query, input_type="search_query")

The asymmetric query/document distinction is, in practice, the biggest reason teams switch from OpenAI to Cohere for retrieval-heavy applications — not raw benchmark numbers, but the fact that the API forces you to think correctly about the two roles a piece of text can play. It's a design choice, not just a model choice, and it tends to produce noticeably better recall in production support-search and document-QA systems.

Open-source embeddings: control, cost, and a widening quality gap that's closing

The open-source embedding ecosystem — think the BGE family, E5, Nomic Embed, GTE, and their many fine-tuned derivatives — has gone from "good enough for a demo" to "genuinely competitive for production" over the last two years. The reason to consider self-hosting isn't just cost savings, though those are real; it's control.

Why self-host at all:

Data never leaves your infrastructure. For healthcare, legal, or financial RAG systems, this alone can be the deciding factor over every other consideration.
No per-token billing. Once you own the GPU (or rent it hourly), embedding a million documents costs the same whether you do it once or ten times.
Fine-tuning is possible. You can continue training an open-source embedding model on your own query-document pairs — something no hosted API lets you do to its base model. A model fine-tuned on your actual support tickets and actual user queries will often beat a stronger general-purpose model that's never seen your domain.
No vendor lock-in on latency or uptime. Your embedding pipeline doesn't go down because a third-party API had an incident.

The real costs:

You now own GPU provisioning, batching, model versioning, and monitoring — work that used to be someone else's problem.
Open-source models generally need more careful chunking and pooling strategy tuning than hosted APIs, which handle a lot of that internally.
Multilingual and long-context support vary a lot model to model; you have to check, not assume.

from sentence_transformers import SentenceTransformer

# BGE and E5-style models often expect task-specific prefixes for best results
model = SentenceTransformer("BAAI/bge-large-en-v1.5")

def embed_open_source(texts, is_query=False):
    prefix = "Represent this sentence for searching relevant passages: " if is_query else ""
    prefixed = [prefix + t for t in texts]
    return model.encode(prefixed, normalize_embeddings=True)

query_vec = embed_open_source(["cancel my plan"], is_query=True)
doc_vecs = embed_open_source(
    ["Subscription cancellation must be requested in writing 30 days prior."],
    is_query=False,
)

Notice the prefix convention — this is a detail that trips up a lot of teams moving from hosted APIs to open-source models. BGE and E5-family models were trained with specific instruction prefixes baked into their training data, and skipping them silently degrades retrieval quality without throwing any error. Always read a model's card on Hugging Face before deploying it; the difference between "works great" and "mysteriously mediocre" is often one missing prefix string.

A direct comparison, feature by feature

Rather than a benchmark table (which goes stale within a quarter and varies wildly by domain), here's how the three camps actually differ on the dimensions that matter for a production decision:

Setup effort: OpenAI and Cohere require an API key and a few lines of code. Open-source models require GPU provisioning, model downloads, and a serving layer (vLLM, TEI, or a simple Flask wrapper) — a day of work minimum, more if you need to scale it.
Cost model: OpenAI and Cohere charge per token embedded, forever. Open-source has an upfront/ongoing compute cost but no per-request billing, which favors high-volume steady workloads and disfavors spiky, low-volume ones.
Data privacy: Open-source wins outright — nothing leaves your network. OpenAI and Cohere both offer enterprise data-handling agreements, but the data still transits a third party's servers.
Retrieval-specific tuning: Cohere's asymmetric query/document input types are the most retrieval-native design of the three. OpenAI's embeddings are general-purpose and solid but not retrieval-specialized. Open-source models vary — some (like the E5 and BGE families) were explicitly trained with retrieval prefixes; others were not.
Fine-tuning on your data: Only open-source models allow this without going through a vendor's fine-tuning program (which, as of 2026, exists for some hosted models but adds cost and complexity).
Multilingual quality: Cohere and select open-source multilingual models (like multilingual-E5) are generally stronger here than OpenAI's default embeddings, though this gap narrows with each model release.
Vector dimensionality and storage: All three now support dimension truncation or native compressed formats (Matryoshka truncation for OpenAI, int8/binary for Cohere, and quantization tooling for most open-source models via libraries like sentence-transformers or FAISS), so this is less of a differentiator than it was two years ago.
Latency: Hosted APIs add network round-trip time per batch; self-hosted models on a nearby GPU can be faster for high-throughput indexing jobs, but hosted APIs typically autoscale better for unpredictable query traffic.

How to actually test embedding models on your own data

Benchmark leaderboards like MTEB are useful for narrowing a shortlist, but they measure aggregate performance across dozens of unrelated tasks and datasets that almost certainly don't resemble your corpus. The only test that matters is one you run yourself, on your own documents and your own real (or realistically simulated) queries.

A minimal evaluation harness looks like this:

Collect 50-100 real questions your users have actually asked (support tickets, search logs, Slack questions — whatever exists).
For each question, hand-label which document chunk(s) should be retrieved as the correct answer.
Embed your document corpus and your test questions with each candidate model.
For each question, retrieve the top-k chunks by cosine similarity and check whether the labeled correct chunk appears in that top-k.
Report recall@5 and recall@10 per model, on your own data, not someone else's benchmark.

import numpy as np

def recall_at_k(query_vecs, doc_vecs, relevant_doc_indices, k=5):
    hits = 0
    for i, qv in enumerate(query_vecs):
        sims = doc_vecs @ qv / (np.linalg.norm(doc_vecs, axis=1) * np.linalg.norm(qv))
        top_k = np.argsort(-sims)[:k]
        if relevant_doc_indices[i] in top_k:
            hits += 1
    return hits / len(query_vecs)

# Run this same function against OpenAI, Cohere, and open-source vectors
# for the same query/document set, and compare the numbers directly.

This takes an afternoon and will tell you more than any published benchmark ever will, because it's measured against the exact distribution of language your system will actually see in production. A model that's ranked lower on MTEB but wins recall@5 on your support-ticket corpus is the correct choice for you, full stop.

Common mistakes teams make when choosing an embedding model

A few patterns show up again and again in RAG systems that underperform, and almost none of them are about picking the "wrong" model in an absolute sense — they're about mismatches between the model and how it's used.

Mixing embedding models across an index. If you embed your documents with one model and later switch to another for new documents without re-embedding everything, similarity search silently breaks — vectors from different models aren't comparable, even if they're the same dimension.
Ignoring the query/document asymmetry. Treating a five-word user query and a 500-word document chunk identically at embedding time throws away signal, especially with models (like Cohere's) built to exploit that distinction.
Chasing MTEB rank instead of testing your own recall. Covered above, but worth repeating: the leaderboard is a starting filter, not a final answer.
Skipping model-specific prefixes or instructions. Many open-source models need specific prefix strings baked in during training; skipping them degrades quality without any visible error.
Re-embedding too rarely. Embedding models improve fast. A pipeline built on a two-year-old model is leaving retrieval quality on the table for the cost of an afternoon's re-indexing job.
Over-indexing on dimensionality. A bigger vector isn't automatically better retrieval — it's more storage and slower search for a benefit that needs to be measured, not assumed.

Chunking and pooling interact with your embedding choice more than people expect

Picking a model is only half the equation — how you chunk your source documents and how the model pools token-level representations into a single vector both interact with the model you choose in ways that aren't obvious until you've been burned by them once.

Most embedding models have an effective context window past which quality degrades even if the API technically accepts more tokens. OpenAI's -3 models and Cohere's embed-v4 both handle a few thousand tokens gracefully, but a single embedding vector representing an entire 3,000-word document tends to be a blurry average of many different ideas — great for coarse topic clustering, poor for precise retrieval. This is why RAG systems chunk documents into paragraph-or-smaller pieces before embedding: you want each vector to represent one coherent idea, not a document's worth of them.

Pooling strategy is the less-discussed cousin of this problem. Most sentence-transformer-style open-source models produce a vector per token internally and then pool them (mean pooling, CLS-token pooling, or last-token pooling) into a single sentence-level embedding. Hosted APIs handle this internally and you never see it, but if you're loading a raw open-source checkpoint yourself, using the wrong pooling method for that specific model — say, mean pooling a model trained with CLS-token pooling — produces embeddings that look plausible but retrieve poorly. This is exactly the kind of silent failure that makes embedding bugs so painful to debug: nothing throws an exception, your vectors have the right shape, and retrieval is just quietly worse than it should be.

# Wrong: assuming mean pooling works for every model
def mean_pool(token_embeddings, attention_mask):
    mask = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    summed = (token_embeddings * mask).sum(1)
    counted = mask.sum(1).clamp(min=1e-9)
    return summed / counted

# Always check the model card for the pooling method it was trained with.
# sentence-transformers models expose this directly:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-large-en-v1.5")
print(model)  # prints the Pooling module config, including pooling_mode

The practical takeaway: when you evaluate a new embedding model, don't just swap the model name and keep everything else the same. Check whether your chunk size is still appropriate for that model's context handling, and if you're self-hosting, verify the pooling method matches what the model card specifies. A recall@5 test that ignores this will quietly penalize a model that would otherwise perform well, and you'll draw the wrong conclusion from your own evaluation.

Hybrid and multi-model strategies worth considering

Nothing forces you to pick exactly one embedding model for an entire system. Several patterns have become common enough in production RAG stacks by 2026 that they're worth knowing before you commit to a single-model architecture.

Dense plus sparse retrieval. Pairing a dense embedding model with a sparse method like BM25 or SPLADE and combining scores (often called hybrid search) catches exact keyword matches — part numbers, error codes, proper nouns — that dense embeddings alone sometimes blur together. Most production vector databases (Postgres with pgvector alongside full-text search, Weaviate, Qdrant) support this natively now, so it's largely a configuration decision rather than an engineering project.
Embed-then-rerank. Use a fast, cheap embedding model to retrieve a wide candidate set (say, top 50), then apply a slower, more precise reranking model (Cohere's rerank models or a cross-encoder) to reorder just those 50 into a final top 5. This gets you most of the precision of an expensive model without paying its cost on your entire corpus.
Different models for different content types. A codebase-heavy knowledge base and a customer-support-ticket knowledge base don't necessarily want the same embedding model. Some teams run a code-specialized embedding model for one index and a general-purpose or support-tuned model for another, then merge results at query time.
Fallback chains. If a hosted API has an outage or a rate-limit spike, having a lightweight open-source model as an emergency fallback keeps search functional, even in degraded form, instead of failing the whole request.

None of these are mandatory for a first version of a RAG system — a single well-chosen embedding model handles the overwhelming majority of use cases fine. But knowing these patterns exist means you're not stuck rebuilding your retrieval layer from scratch the day a single model stops being good enough.

Putting it together: a practical decision framework

If you're starting a new RAG project today and need a default: OpenAI's `text-embedding-3-small` is a completely reasonable first choice — cheap, fast, well-documented, and good enough that you should spend your engineering time on chunking and reranking before you spend it re-litigating the embedding model.

If retrieval precision is the product — a legal research tool, a support-deflection bot judged on ticket-resolution rate, a multilingual search feature — Cohere's `embed-v4` family is worth the switch specifically because of the query/document asymmetry and multilingual strength. Run the recall@k test above before committing; the gain is real but you should see it on your own data first.

If data residency is non-negotiable, if you're running at a volume where per-token billing becomes a real cost center, or if you have the labeled query-document pairs to fine-tune on your own domain, self-hosted open-source models (BGE, E5, Nomic Embed, or their successors) are the right investment. Budget real engineering time for serving infrastructure and prefix/pooling correctness — the model quality is there, but the operational surface area is larger than a hosted API call.

None of these choices are permanent. Embedding models are cheap to swap relative to the rest of a RAG stack — as long as you re-embed your entire corpus when you switch, rather than mixing vectors from two different models in one index. Treat the embedding model as a component you'll revisit every six to twelve months as the field moves, not a decision you make once and forget.

If you're still fuzzy on how embeddings fit into the bigger retrieval-augmented generation picture — chunking strategy, vector databases, reranking, and how it all connects to the generation step — that foundational context is exactly what we cover in Introduction to RAG on teachyou.ai, and it's the natural next step before you commit to a model for production.