EmbeddingsRAGfine-tuningsemantic searchsentence-transformers

Fine-Tuning Embedding Models for Your Domain

Pramod Dutta · Jul 3, 2026 · 14 min read

Embedding fine-tuning is the process of taking a pretrained text embedding model and continuing its training on pairs or triplets from your own domain, so the vector space it produces separates your documents the way your users actually search for them. Most teams reach for a general-purpose embedding model, wire it into a RAG pipeline, and accept whatever recall they get. Embedding fine-tuning is what you do when that recall plateaus below what the business needs and reranking or chunking changes stop moving the number.

This article walks through when fine-tuning is worth the effort, how to build a dataset without hand-labeling thousands of examples, and a full training and evaluation loop using sentence-transformers. Everything below assumes you already have a working RAG or search pipeline and are trying to improve retrieval quality, not build one from scratch.

Why General-Purpose Embeddings Miss Your Domain

Public embedding models are trained on broad web text: Wikipedia, forums, news, code comments, academic abstracts. They learn that "car" and "automobile" are close, that "python" the language and "python" the snake are different senses depending on context, and that questions and their answers should land near each other in vector space. None of that training data has seen your internal ticket taxonomy, your product's SKU naming convention, your legal team's contract clauses, or the shorthand your support agents use in Slack.

The failure mode is usually silent. The model doesn't error out, it just returns plausible-looking but wrong neighbors. A query for "reset MFA on a shared device" might retrieve a generic "reset password" article instead of the specific shared-device MFA runbook, because the model has never seen enough examples where those two phrasings needed to diverge. In legal, medical, and other jargon-heavy domains this gets worse: "consideration" means something very different in a contract than in casual English, and a general embedding model will not have learned that distinction from web text.

Embedding fine-tuning fixes this by showing the model thousands of examples of what "close" and "far apart" mean specifically in your data. You are not teaching it new facts, you are reshaping the geometry of its vector space around your domain's actual similarity judgments.

When Fine-Tuning Is the Right Move (and When It Isn't)

Fine-tuning an embedding model is not the first lever to pull. Before you invest in it, rule out the cheaper fixes:

Bad chunking. If your chunks mix unrelated topics or are too large, no embedding model will save you. Fix chunk boundaries first.
No reranker. A cross-encoder reranker on top of a general embedding model's top-k results often closes most of the recall gap for a fraction of the effort of fine-tuning.
Wrong base model. Some embedding models are simply stronger than others for your language or content type. Swapping models is a one-line change; try it before training one.
Hybrid search missing. Combining dense vector search with BM25 keyword search catches exact-match cases (product codes, error strings, IDs) that pure embeddings routinely miss.

Fine-tuning earns its cost when you've done the above and retrieval still misses on domain-specific vocabulary, near-duplicate documents that differ in one critical clause, or queries phrased the way your users phrase them rather than the way a general web corpus phrases them. It also pays off when you have enough real usage data (click logs, support ticket resolutions, "was this helpful" signals) to build a training set without paying for large-scale manual labeling.

If you have fewer than a few hundred good positive pairs and no way to generate more, fine-tuning is unlikely to help and may overfit. That's the threshold to keep in mind before committing engineering time.

Building a Training Dataset for Embedding Fine-Tuning

The core unit of data for embedding fine-tuning is a pair: a query (or anchor text) and a document that should be considered relevant to it. Most modern training setups use one of three shapes:

Positive pairs: (query, relevant_document). Works with in-batch negatives, where every other document in the batch is treated as a negative for this query.
Triplets: (anchor, positive, negative). You explicitly supply a document that should NOT match, which is more sample-efficient than relying on random in-batch negatives.
Labeled pairs with a score: (text_a, text_b, similarity_score). Used with cosine similarity loss when you have graded relevance rather than binary.

Where the data comes from matters more than the loss function you pick. Good sources, roughly in order of quality:

Search logs with click-through or dwell time. A query followed by a clicked result and a long session is a strong positive signal.
Support ticket to knowledge-base article mappings. If your support tool logs which article an agent attached to resolve a ticket, that's a labeled pair for free.
FAQ and documentation Q&A pairs. If you already have a help center, the question is the anchor and the answer section is the positive.
LLM-generated synthetic queries. Feed a document to an LLM and ask it to generate 3-5 realistic queries a user might type to find it. This is the fastest way to bootstrap a dataset when logs don't exist yet.

Here's a script that generates synthetic query-document pairs from a folder of documents using an LLM, which is often the fastest path to a first training set:

import json
import os
from pathlib import Path
from anthropic import Anthropic

client = Anthropic()

PROMPT = """Given this document, write 4 short, realistic search queries
a user might type to find it. Vary phrasing: some casual, some precise,
some using synonyms instead of the document's exact wording.
Return one query per line, no numbering, no extra text.

Document:
{doc}
"""

def generate_queries(doc_text: str) -> list[str]:
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=300,
        messages=[{"role": "user", "content": PROMPT.format(doc=doc_text[:3000])}],
    )
    text = response.content[0].text
    return [line.strip() for line in text.splitlines() if line.strip()]

def build_dataset(docs_dir: str, out_path: str):
    pairs = []
    for path in Path(docs_dir).glob("*.txt"):
        doc_text = path.read_text()
        for query in generate_queries(doc_text):
            pairs.append({"query": query, "positive": doc_text})
    with open(out_path, "w") as f:
        for pair in pairs:
            f.write(json.dumps(pair) + "\n")

if __name__ == "__main__":
    build_dataset("docs/", "train_pairs.jsonl")

For negatives, don't just pick random documents. Random negatives are too easy and the model learns almost nothing from them once training progresses past the first epoch. Instead, mine hard negatives: documents that are topically close but not actually relevant. A simple way to find them without any extra tooling is to run your current (unfine-tuned) embedding model over the corpus, retrieve the top 10 results for each training query, and take documents ranked 3 through 8 that are not the known positive as hard negative candidates. They're similar enough to be genuinely confusing, which is exactly what you want the model to learn to separate.

def mine_hard_negatives(query, positive_doc, corpus_embeddings, corpus_docs, model, k=8):
    query_embedding = model.encode(query)
    scores = corpus_embeddings @ query_embedding
    top_k_idx = scores.argsort()[::-1][:k]
    candidates = [corpus_docs[i] for i in top_k_idx if corpus_docs[i] != positive_doc]
    return candidates[2:5]  # skip the very top matches, they may be near-duplicates of the positive

Aim for at least 1,000-2,000 pairs before fine-tuning a small model, and several thousand if you're fine-tuning a larger base model. Below that, in-batch negative training with a large batch size can still work reasonably well because it manufactures extra negatives from every other example in the batch.

Choosing a Base Model and Framework

sentence-transformers is the standard library for embedding fine-tuning and works with most open embedding model families. Pick a base model with these criteria:

Matches your deployment constraints. A smaller model (under 500M parameters) is cheaper to run and fine-tunes faster; a larger one has more capacity to learn nuanced domain distinctions but costs more at inference time.
Already strong in your language and domain neighborhood. Fine-tuning improves a model, it doesn't fix a fundamentally wrong starting point. Check a general embedding benchmark leaderboard for your language before committing.
Compatible dimensionality with your vector database. Changing embedding dimension means reindexing everything, which is a bigger operation than the fine-tune itself.

The loss function that works best for most retrieval fine-tuning is MultipleNegativesRankingLoss (sometimes called in-batch negatives loss or InfoNCE). It treats every positive pair in a training batch as a negative for every other pair, so a batch of 32 pairs effectively gives you 31 negatives per example for free, on top of any hard negatives you supply explicitly.

Fine-Tuning Walkthrough

Here is a complete, runnable fine-tuning script using sentence-transformers. Install the library first:

pip install sentence-transformers datasets

import json
from sentence_transformers import (
    SentenceTransformer,
    InputExample,
    losses,
    evaluation,
)
from torch.utils.data import DataLoader

def load_pairs(path):
    examples = []
    with open(path) as f:
        for line in f:
            row = json.loads(line)
            examples.append(InputExample(texts=[row["query"], row["positive"]]))
    return examples

def load_triplets(path):
    examples = []
    with open(path) as f:
        for line in f:
            row = json.loads(line)
            examples.append(
                InputExample(texts=[row["query"], row["positive"], row["negative"]])
            )
    return examples

# 1. Load a base model
model = SentenceTransformer("BAAI/bge-base-en-v1.5")

# 2. Load training data (use triplets if you mined hard negatives, else pairs)
train_examples = load_triplets("train_triplets.jsonl")
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=32)

# 3. Pick a loss. MultipleNegativesRankingLoss handles both pair and triplet input.
train_loss = losses.MultipleNegativesRankingLoss(model)

# 4. Build a small held-out evaluator to track progress during training
with open("eval_pairs.jsonl") as f:
    eval_rows = [json.loads(line) for line in f]

evaluator = evaluation.InformationRetrievalEvaluator(
    queries={str(i): row["query"] for i, row in enumerate(eval_rows)},
    corpus={str(i): row["positive"] for i, row in enumerate(eval_rows)},
    relevant_docs={str(i): {str(i)} for i in range(len(eval_rows))},
    name="domain-eval",
)

# 5. Fine-tune
model.fit(
    train_objectives=[(train_dataloader, train_loss)],
    evaluator=evaluator,
    epochs=4,
    warmup_steps=100,
    evaluation_steps=200,
    output_path="models/domain-embedder-v1",
    save_best_model=True,
)

A few practical settings that matter more than people expect:

Batch size drives negative quality with in-batch losses. Larger batches give more implicit negatives per step. If you have GPU memory, push batch size to 64 or 128 before adding more epochs.
Keep epochs low. Embedding fine-tuning overfits fast, usually within 3-5 epochs on a few thousand pairs. Watch the evaluator score per checkpoint and stop when it plateaus rather than running a fixed schedule blindly.
Freeze nothing by default. Full fine-tuning of the base model works better than freezing early layers for most domain adaptation cases, unless your dataset is very small (under 500 pairs), in which case freezing the bottom half of the transformer layers can reduce overfitting.
Normalize embeddings at inference time, matching whatever normalization the loss function assumed. MultipleNegativesRankingLoss with cosine similarity expects normalized vectors; sentence-transformers handles this automatically if you use model.encode(text, normalize_embeddings=True).

Evaluating the Fine-Tuned Model

Never ship a fine-tuned embedding model on training loss alone. Build a held-out evaluation set that mirrors real usage, and measure retrieval metrics before and after fine-tuning on the exact same set.

The two metrics that matter for retrieval:

Recall@k: of the queries where a known relevant document exists, what fraction have it in the top k retrieved results. This is the number product teams care about, since it tells you whether the right answer would even reach a reranker or LLM.
Mean Reciprocal Rank (MRR): rewards the correct document appearing near the top of the ranking, not just somewhere in the top k.

from sentence_transformers import SentenceTransformer, util
import json

def evaluate_model(model_path, eval_path, k=5):
    model = SentenceTransformer(model_path)
    with open(eval_path) as f:
        rows = [json.loads(line) for line in f]

    corpus = [row["positive"] for row in rows]
    corpus_embeddings = model.encode(corpus, normalize_embeddings=True)

    hits_at_k = 0
    reciprocal_ranks = []

    for i, row in enumerate(rows):
        query_embedding = model.encode(row["query"], normalize_embeddings=True)
        scores = util.cos_sim(query_embedding, corpus_embeddings)[0]
        ranking = scores.argsort(descending=True).tolist()
        rank = ranking.index(i) + 1
        if rank <= k:
            hits_at_k += 1
        reciprocal_ranks.append(1.0 / rank)

    recall_at_k = hits_at_k / len(rows)
    mrr = sum(reciprocal_ranks) / len(reciprocal_ranks)
    return {"recall@k": recall_at_k, "mrr": mrr}

before = evaluate_model("BAAI/bge-base-en-v1.5", "eval_pairs.jsonl")
after = evaluate_model("models/domain-embedder-v1", "eval_pairs.jsonl")
print("Base model:", before)
print("Fine-tuned:", after)

Run this against a general embedding benchmark subset too, not just your domain eval set. It's common for a domain fine-tune to gain 10-20 points of recall on domain queries while quietly losing a few points on generic queries outside the training distribution. Whether that trade-off is acceptable depends on how narrow your production traffic actually is. If your product only ever sees domain queries, the trade is an easy yes. If general queries still show up, weigh both numbers before rolling out.

Deploying the Fine-Tuned Model in a RAG Pipeline

Once the fine-tuned model beats the baseline on your held-out set, the deployment mechanics are the same as swapping any embedding model:

Re-embed your entire corpus with the new model and load it into a fresh index or a fresh namespace/collection in your vector database. Do not mix vectors from two different models in the same index; distances between them are meaningless.
Version the index name, something like docs-embedder-v2, so you can roll back to the previous model's index instantly if a regression shows up in production.
Run a shadow evaluation where the new model's retrieved results are logged alongside the old model's for the same live queries, without switching user-facing traffic yet. Compare which one a human or an LLM judge would rate as more relevant.
Cut over gradually. Route a small percentage of traffic to the new index, watch downstream signals (click-through, "was this helpful," support escalation rate), and expand once the numbers hold.

Keep the fine-tuning script, dataset, and evaluation set under version control together with the model checkpoint. Domain data drifts: new product lines, new terminology, new document types. Plan to refresh the training set and re-run fine-tuning on a cadence (quarterly is a reasonable default) rather than treating this as a one-time project.

Common Pitfalls in Embedding Fine-Tuning

Training on queries that don't resemble production traffic. Synthetic LLM-generated queries are a good bootstrap but skew toward well-formed, grammatically complete sentences. Real users type fragments, misspellings, and abbreviations. Mix in real query logs as soon as you have any.
Skipping hard negative mining. Training only with in-batch random negatives caps how sharp the model's boundaries can get, especially once you're past the first epoch or two.
Overfitting to a small eval set. If your held-out set is the same 50 examples you've been tuning hyperparameters against for a week, the "improvement" you're seeing may not generalize. Rotate or expand the eval set periodically.
Ignoring inference-time normalization mismatches. A model trained with cosine similarity loss but queried with dot product similarity (or vice versa) at inference time will silently underperform without throwing any error.
Re-embedding only new documents. After fine-tuning, the entire corpus needs new vectors, not just newly added documents. Mixed old and new vectors in the same index produce inconsistent retrieval quality that's hard to debug because it looks random.
Forgetting to test on adversarial near-duplicates. If your domain has documents that differ by one critical clause (a contract amendment, a changed pricing tier), make sure some of your training and eval pairs specifically test whether the model can tell them apart, not just whether it can match broad topics.

FAQ

How much data do I actually need to fine-tune an embedding model? A few hundred pairs can produce a measurable improvement if they're high quality and include hard negatives, but a few thousand pairs gives a more reliable and generalizable result. Synthetic query generation from an LLM is the fastest way to get from zero to a usable first dataset.

Should I fine-tune the embedding model or just add a reranker? Try the reranker first. It's cheaper to set up, doesn't require training data curation, and often recovers most of the recall you're missing. Fine-tune the embedding model when you need faster first-stage retrieval quality (rerankers only fix the ordering of whatever the embedding model already retrieved) or when the reranker still isn't enough.

Can I fine-tune a proprietary embedding API model, like the ones from OpenAI or Cohere? Some providers offer managed fine-tuning endpoints for their embedding models; check the specific provider's current documentation, since this capability and its constraints change frequently. Open-weight models via sentence-transformers give you full control over the training process and are the more common path for teams that want to own the resulting model.

How often should I retrain the embedding model? Retrain when you notice retrieval quality degrading on new content categories, or on a fixed cadence like quarterly if your domain vocabulary evolves steadily. Keep collecting real query and click data continuously so each retrain has fresher signal than the last.

Will fine-tuning hurt performance on queries outside my domain? Often slightly, yes. The model's vector space reshapes around your training distribution, which can pull it away from general-purpose behavior it isn't seeing reinforced anymore. Evaluate on a general benchmark subset alongside your domain eval set so you know the size of that trade-off before deploying.

What's the difference between fine-tuning an embedding model and fine-tuning the LLM that reads the retrieved context? They solve different problems. Embedding fine-tuning improves which documents get retrieved in the first place. LLM fine-tuning changes how the model reasons over or phrases answers once it already has the right context. A retrieval quality problem needs the former; a generation quality problem needs the latter. Most RAG systems get more value from fixing retrieval first, since no amount of generation fine-tuning helps if the LLM never sees the right document.