Ragas

Ragas for Multi-Language RAG Systems

Pramod Dutta · May 7, 2026 · 12 min read

Why Multi-Language RAG Breaks Your Evaluation Assumptions

You built a RAG pipeline. It works beautifully on English test questions. Faithfulness scores look great, context precision is solid, and your demo to stakeholders goes off without a hitch. Then someone in your Mumbai office asks a question in Hindi, someone in your Berlin office asks in German, and someone in your Sao Paulo office types a question that mixes Portuguese with English product names. Suddenly your evaluation numbers stop meaning anything, because you never actually measured what happens when the query language, the document language, and the answer language are not all the same thing.

Multi-language RAG is not "RAG but with translation slapped on top." It introduces a whole new axis of failure: retrieval can succeed in the source language but fail to surface the right passage when the query is phrased differently linguistically, generation can produce fluent but factually unfaithful answers because the LLM is loosely paraphrasing across languages, and your evaluation metrics themselves may be silently biased toward whichever language your judge LLM was trained on most heavily.

This is where Ragas earns its keep. Ragas is not just a metrics library — it is a framework for building evaluation pipelines that can be adapted to non-English and cross-lingual scenarios, provided you understand where the defaults will mislead you and where you need to intervene. In this article we will walk through exactly that: what breaks in multi-language RAG, how Ragas's core metrics behave across languages, how to configure LLM-based judges for fairer scoring, and how to build a genuinely useful multi-language evaluation harness with real code.

What "Multi-Language RAG" Actually Means (Three Distinct Scenarios)

Before touching code, it helps to separate three scenarios that get lumped together under "multi-language RAG" but require different evaluation strategies.

Monolingual-per-market: Your knowledge base is in Language A, and users in that market query in Language A. You just happen to run this same pipeline for many languages in parallel (English docs for English users, Japanese docs for Japanese users, etc.). The evaluation challenge here is mostly about judge LLM reliability per language, not cross-lingual retrieval.
Cross-lingual retrieval: Your knowledge base is in one language (often English, because that's where most enterprise documentation lives), but users query in their native language. The retriever must bridge languages, usually via multilingual embeddings. This is the hardest case, and it's where most multi-language RAG systems actually fail — in retrieval, not generation.
Mixed-language corpus: Your knowledge base itself contains documents in multiple languages (support tickets, contracts, product manuals), and a single query might need to pull from documents in two or three different languages to answer well. This is common in global enterprises and is the least discussed in tutorials.

Each of these needs a slightly different evaluation setup. If you only test scenario one and assume it generalizes, you will ship a system that quietly fails scenario two, which is usually the one your business actually needed.

How Ragas's Core Metrics Behave Across Languages

Ragas's headline metrics — faithfulness, answer relevancy, context precision, and context recall — are all computed using an LLM as a judge, which decomposes the answer into claims, checks those claims against retrieved context, and scores relevance. This decomposition step is where language sensitivity creeps in.

Faithfulness asks the judge LLM to break the generated answer into atomic statements and verify each one is supported by the retrieved context. When the answer and context are in different languages (a common outcome in cross-lingual RAG where context is in English but the answer is generated in French), the judge has to perform an implicit translation-and-verify step. Weaker judge models get this wrong more often than they do for same-language verification, because subtle claims can get lost or altered in the implicit translation.

Answer relevancy works by having the judge generate hypothetical questions from the answer and comparing them (via embedding similarity) to the original question. If your embedding model has weaker multilingual coverage than your judge LLM, this metric degrades even when the answer is genuinely relevant, because the embedding space just doesn't align well across the two languages involved.

Context precision and recall depend on the judge correctly identifying whether a retrieved chunk is useful for answering the question. This is usually more robust across languages than faithfulness, because it's a coarser yes/no judgment rather than fine-grained claim verification — but it still degrades for low-resource languages where the judge LLM has seen less training data.

The practical takeaway: don't assume a faithfulness score of 0.85 means the same thing in Vietnamese that it means in English. You need per-language baselines, not one global threshold.

Setting Up a Multi-Language Ragas Evaluation

Let's build this concretely. The key decisions are: which LLM judges the outputs, which embedding model measures semantic similarity, and how you structure your evaluation dataset so language is a tracked variable, not an afterthought.

from ragas import evaluate, EvaluationDataset
from ragas.metrics import (
    Faithfulness,
    AnswerRelevancy,
    LLMContextPrecisionWithReference,
    LLMContextRecall,
)
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# Use a strong multilingual-capable judge model.
# Model choice matters more here than in English-only evaluation.
judge_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o", temperature=0))

# Use an embedding model with genuine multilingual training,
# not one that was only fine-tuned on English retrieval pairs.
multilingual_embeddings = LangchainEmbeddingsWrapper(
    OpenAIEmbeddings(model="text-embedding-3-large")
)

metrics = [
    Faithfulness(llm=judge_llm),
    AnswerRelevancy(llm=judge_llm, embeddings=multilingual_embeddings),
    LLMContextPrecisionWithReference(llm=judge_llm),
    LLMContextRecall(llm=judge_llm),
]

Notice that both the LLM and the embeddings are passed explicitly. Relying on Ragas's defaults silently locks you into whatever the library ships with, which may not be the best choice for your target languages. This single change — being deliberate about judge and embedding selection — fixes a large share of "my multi-language scores look wrong" issues.

Structuring Your Evaluation Dataset by Language

The single biggest mistake teams make is dumping every language into one evaluation run and reporting a blended average. A blended average hides exactly the information you need: which languages are underperforming and why. Structure your dataset so language is a first-class field.

import pandas as pd
from ragas import EvaluationDataset

# Each row should carry a language tag alongside the standard
# Ragas fields (user_input, retrieved_contexts, response, reference)
rows = [
    {
        "user_input": "Quelle est la politique de remboursement ?",
        "retrieved_contexts": ["Les remboursements sont traités sous 14 jours..."],
        "response": "Les remboursements sont traités sous 14 jours ouvrables.",
        "reference": "Refunds are processed within 14 business days.",
        "language": "fr",
    },
    {
        "user_input": "返品ポリシーは何ですか？",
        "retrieved_contexts": ["Refunds are processed within 14 business days..."],
        "response": "返金は14営業日以内に処理されます。",
        "reference": "Refunds are processed within 14 business days.",
        "language": "ja",
    },
    {
        "user_input": "What is the refund policy?",
        "retrieved_contexts": ["Refunds are processed within 14 business days..."],
        "response": "Refunds are processed within 14 business days.",
        "reference": "Refunds are processed within 14 business days.",
        "language": "en",
    },
]

df = pd.DataFrame(rows)
dataset = EvaluationDataset.from_pandas(df)

result = evaluate(dataset=dataset, metrics=metrics)
result_df = result.to_pandas()

# Now break results down by language — this is the step
# most teams skip, and it's the one that actually matters.
result_df["language"] = df["language"]
per_language = result_df.groupby("language")[
    ["faithfulness", "answer_relevancy", "llm_context_precision_with_reference", "context_recall"]
].mean()

print(per_language)

Running this consistently will surface patterns you'd otherwise miss: maybe your Japanese faithfulness scores are systematically lower not because the answers are worse, but because your judge LLM is stricter about claim decomposition in Japanese due to how sentence boundaries work. That's a calibration issue, not a real quality gap, and you only find it by segmenting.

Handling Cross-Lingual Retrieval Failures Before They Reach Generation

A large share of multi-language RAG complaints ("the answer is wrong") are actually retrieval failures in disguise. The generator did a fine job answering the context it was given — the context was just wrong, because the retriever couldn't bridge the query language to the document language.

Ragas's context precision and recall metrics are your diagnostic tool here, but you need a reference answer to compute recall meaningfully. If you don't have references yet, start with precision alone, since it only needs the question and retrieved contexts.

from ragas.metrics import LLMContextPrecisionWithoutReference

# Useful for a quick retrieval-only health check when you
# don't yet have gold reference answers for every language.
retrieval_only_metrics = [
    LLMContextPrecisionWithoutReference(llm=judge_llm),
]

retrieval_result = evaluate(dataset=dataset, metrics=retrieval_only_metrics)
retrieval_df = retrieval_result.to_pandas()
retrieval_df["language"] = df["language"]

low_precision = retrieval_df[retrieval_df["llm_context_precision_without_reference"] < 0.5]
print(f"Languages with weak retrieval: {low_precision['language'].unique()}")

If you find that precision craters specifically for cross-lingual pairs (query in Language A, documents in Language B) but stays healthy for monolingual pairs, that's a strong signal your embedding model's cross-lingual alignment is the bottleneck, not your generator or your judge. This distinction changes what you fix: swap the embedding model or add query translation before retrieval, rather than tweaking prompts on the generation side.

A Practical Fix: Query Translation as a Retrieval Bridge

One pattern that consistently improves cross-lingual retrieval scores is translating the query into the corpus's primary language before embedding it for search, then generating the final answer back in the user's language. This adds latency but often meaningfully improves context precision and recall, which you can verify directly with Ragas before and after.

from langchain_openai import ChatOpenAI

translator = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def translate_query(query: str, target_lang: str = "English") -> str:
    prompt = (
        f"Translate the following question into {target_lang}. "
        f"Return only the translated text, nothing else.\n\n{query}"
    )
    return translator.invoke(prompt).content.strip()

def retrieve_with_bridge(query: str, retriever, corpus_language: str = "English"):
    bridged_query = translate_query(query, target_lang=corpus_language)
    return retriever.get_relevant_documents(bridged_query)

Run your Ragas context precision evaluation once with direct retrieval and once with the translation bridge, on the same held-out set of cross-lingual questions. The delta tells you, in concrete numbers, whether the extra translation hop is worth the added latency and cost for your use case. Don't guess — measure it.

Building Custom Metrics for Language-Specific Quality Checks

Sometimes the built-in metrics don't capture what actually matters for a specific language pair. A common example: in customer support RAG for Japanese or Korean, formality register (keigo, honorifics) matters as much as factual accuracy, but none of Ragas's default metrics touch tone. Ragas supports custom metrics built on its MetricWithLLM base, which lets you write exactly this kind of check.

from ragas.metrics.base import MetricWithLLM, SingleTurnMetric
from ragas.prompt import PydanticPrompt
from pydantic import BaseModel
import typing as t

class FormalityInput(BaseModel):
    response: str
    language: str

class FormalityOutput(BaseModel):
    is_appropriately_formal: bool
    reasoning: str

class FormalityPrompt(PydanticPrompt[FormalityInput, FormalityOutput]):
    instruction = (
        "Given a customer support response and its language, judge whether "
        "the formality register is appropriate for professional customer "
        "support in that language and culture. Consider honorifics, "
        "politeness markers, and tone."
    )
    input_model = FormalityInput
    output_model = FormalityOutput

class FormalityCheck(MetricWithLLM, SingleTurnMetric):
    name: str = "formality_check"

    async def _single_turn_ascore(self, sample, callbacks) -> float:
        prompt = FormalityPrompt()
        result = await prompt.generate(
            data=FormalityInput(response=sample.response, language=sample.language),
            llm=self.llm,
            callbacks=callbacks,
        )
        return 1.0 if result.is_appropriately_formal else 0.0

This is a small amount of code, but it plugs directly into the same evaluate() call as the built-in metrics, giving you a unified report that includes both universal RAG quality signals (faithfulness, relevancy) and language-specific ones (formality, terminology consistency, script correctness) side by side.

Common Pitfalls Teams Hit With Multi-Language Ragas Evaluation

A few patterns show up repeatedly when teams adopt Ragas for multi-language systems, worth naming explicitly so you can check for them in your own setup.

Using an English-centric judge LLM without verification: Not every model judges every language equally well. Spot-check judge outputs against native-speaker review for at least your top three non-English languages before trusting the automated scores.
Reusing English prompt templates verbatim in evaluation: If your custom metrics use PydanticPrompt classes with instruction text written for English nuance, verify the instructions still make sense when the response field being judged is in a different language. Instructions themselves usually stay in English, since that's what the judge model reasons in, but double-check assumptions baked into the wording (e.g., referencing "sentences" when the target language doesn't use sentence boundaries the same way).
Ignoring script and encoding issues in chunking: If your document loader or text splitter was tuned on Latin-script text, it can silently mis-chunk CJK or right-to-left scripts, which shows up in evaluation as low context precision that looks like a retrieval problem but is actually a preprocessing bug.
Averaging across languages in dashboards: As covered above, always keep language as a groupby key in your reporting, never a blended metric.
Skipping cost planning: LLM-judged evaluation across many languages multiplies your judge LLM calls by however many languages you test. Budget for this explicitly, and consider using a cheaper judge model for CI regression checks while reserving your strongest judge for periodic deep audits.

Putting It Together: A Repeatable Multi-Language Evaluation Loop

The goal is not a one-time evaluation run but a repeatable loop you run every time you change your retriever, your embedding model, your prompt, or your corpus. A practical loop looks like this:

Maintain a held-out evaluation set with balanced representation across your target languages, tagged by language and by scenario type (monolingual, cross-lingual, mixed-corpus).
Run the full Ragas metric suite (faithfulness, answer relevancy, context precision, context recall) plus any custom metrics specific to your domain, with an explicitly chosen judge LLM and embedding model.
Segment every result by language before drawing conclusions. Never trust a single blended number.
When a language underperforms, isolate whether the failure is in retrieval (check context precision/recall) or generation (check faithfulness/relevancy) before changing anything.
Re-run the same evaluation set after each fix to confirm the change actually moved the needle for that specific language, not just the average.
Periodically spot-check judge outputs with native speakers, since automated judges can drift or be systematically biased for languages they were less exposed to during training.

This loop is simple to describe but easy to skip under deadline pressure, which is exactly when multi-language quality problems slip into production and surface as complaints from your international users months later.

Closing Thoughts

Multi-language RAG evaluation is not a bolt-on to your existing English evaluation pipeline — it requires deliberate choices about judge models, embedding models, dataset structure, and custom metrics, plus discipline about segmenting results by language instead of averaging them away. Ragas gives you the building blocks: configurable LLMs and embeddings, a clean metric interface, and a straightforward path to custom metrics for whatever your specific languages and domains demand. The rest is process — building the evaluation set, running it consistently, and actually reading the per-language breakdown before you ship.

If you want to go deeper on building these evaluation pipelines hands-on, including cross-lingual retrieval debugging, custom metric design, and CI integration for RAG systems, check out the Ragas Tutorial course on teachyou.ai, where we build a full multi-language evaluation harness from scratch and break down exactly where real production pipelines tend to fail.