Ragas Integration with LangChain: A Practical Guide

Why your LangChain RAG pipeline needs Ragas

You built a retrieval-augmented generation pipeline with LangChain. It retrieves chunks, stuffs them into a prompt, and the LLM answers. It looks fine in your demo. Then it goes to production, and within a week someone in support forwards you a screenshot where the bot confidently answered a question with information that was nowhere in the retrieved context. Nobody flagged it until a customer did.

This is the central problem with RAG systems: they fail silently. A retriever can pull the wrong chunks and the generator will happily write a fluent, wrong answer on top of them. Unlike a classifier where you can eyeball a confusion matrix, a RAG pipeline's failure modes are split across two systems — the retriever and the generator — and a bad output could be either one's fault, or both.

Ragas (Retrieval Augmented Generation Assessment) exists to make this measurable. It gives you a set of metrics — faithfulness, answer relevance, context precision, context recall, and more — that quantify exactly where a RAG pipeline is weak. Instead of "the answer feels off," you get "faithfulness score dropped to 0.42 on this batch, which means the model is hallucinating claims not grounded in retrieved context."

LangChain is the most common framework people use to build the RAG pipeline in the first place, so a natural question is: how do you get Ragas and LangChain talking to each other without rewriting your whole pipeline? That is what this guide walks through — end to end, with runnable code, not just theory.

We'll build a small LangChain RAG pipeline, generate a test set, run it through Ragas metrics, and wire the whole thing into a CI-style evaluation script you can actually keep using after you close this tab.

Setting up the environment

Before touching any code, get your dependencies straight. Ragas is designed to sit on top of LangChain's abstractions (LLM, Embeddings, Document), which is exactly why the integration is smoother than it looks at first glance.

pip install langchain langchain-openai langchain-community ragas datasets faiss-cpu

A few notes on versions that matter in practice:

• Ragas expects an OpenAI-compatible LLM and embeddings object, but it doesn't require OpenAI specifically. It will happily wrap a LangChain ChatOpenAI, AzureChatOpenAI, or any LangChain-compatible chat model through its LangchainLLMWrapper.
• You need datasets because Ragas evaluation objects are backed by a Hugging Face Dataset under the hood.
• faiss-cpu is just for this tutorial's vector store — swap in Chroma, Pinecone, or Qdrant in your own pipeline without changing anything downstream in the Ragas layer.

Set your API key before running anything:

export OPENAI_API_KEY="sk-..."

If you're using Azure OpenAI or a self-hosted model behind an OpenAI-compatible endpoint, the same wrapper pattern applies — you just configure the LangChain chat model differently and pass it into Ragas the same way.

Building a minimal LangChain RAG pipeline

To evaluate a RAG pipeline, you need one first. Here's a compact pipeline: load documents, chunk them, embed them, retrieve, and generate an answer with a retrieval chain.

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import FAISS
from langchain.chains import RetrievalQA

# 1. Load your source documents
loader = TextLoader("docs/product_manual.txt")
documents = loader.load()

# 2. Split into chunks
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(documents)

# 3. Embed and index
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = FAISS.from_documents(chunks, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

# 4. Build the RAG chain
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True,
)

response = qa_chain.invoke({"query": "What is the warranty period for the device?"})
print(response["result"])
print([doc.page_content[:80] for doc in response["source_documents"]])

This is deliberately simple. RetrievalQA is a legacy-style chain, but it's still the fastest way to demonstrate the pattern, and everything here maps directly onto LCEL (LangChain Expression Language) pipelines if that's what you're using in production. The important part for Ragas is that you have three things coming out of every query: the question, the generated answer, and the retrieved context documents.

Understanding what Ragas actually measures

Before writing evaluation code, it helps to know what each metric is actually checking, because picking the wrong metric for your failure mode wastes a lot of debugging time.

• Faithfulness — does the generated answer only contain claims that are supported by the retrieved context? This is your hallucination detector. A low score means the LLM is inventing facts not present in what was retrieved.
• Answer relevancy — does the answer actually address the question asked, regardless of whether it's grounded? A high-faithfulness, low-relevancy answer is one that's accurate but off-topic or evasive.
• Context precision — of the chunks retrieved, how many were actually useful for answering the question? This flags a noisy retriever pulling irrelevant chunks alongside good ones.
• Context recall — did the retriever pull *all* the information needed to answer correctly? This requires a ground-truth reference answer, and it flags a retriever that's missing critical chunks entirely.

Notice that faithfulness and answer relevancy evaluate the generator, while context precision and context recall evaluate the retriever. This split matters enormously in practice — if your faithfulness score is low, tuning your retriever won't fix it; you need to fix your prompt or your generation temperature. If your context recall is low, no amount of prompt engineering will save you; you need better chunking or a better retriever.

Ragas also ships newer metrics like answer_correctness and answer_similarity for cases where you have a gold-standard reference answer and want a direct comparison score, and context_entity_recall for domain-heavy content where entity coverage (names, part numbers, dates) matters more than prose similarity.

Wiring LangChain outputs into a Ragas dataset

Ragas expects your evaluation data in a specific shape: a Hugging Face Dataset with columns question, answer, contexts, and optionally ground_truth. Here's how to go from your LangChain chain's raw output to that shape.

from datasets import Dataset

questions = [
    "What is the warranty period for the device?",
    "Can the battery be replaced by the user?",
    "What voltage range does the charger support?",
]

ground_truths = [
    "The device comes with a 2-year limited warranty.",
    "No, the battery is sealed and must be replaced by an authorized service center.",
    "The charger supports 100V to 240V input.",
]

answers = []
contexts = []

for question in questions:
    result = qa_chain.invoke({"query": question})
    answers.append(result["result"])
    contexts.append([doc.page_content for doc in result["source_documents"]])

eval_dataset = Dataset.from_dict({
    "question": questions,
    "answer": answers,
    "contexts": contexts,
    "ground_truth": ground_truths,
})

This loop is the actual integration point between LangChain and Ragas — there's no special adapter object required. You just run your existing chain, collect question, answer, and contexts (a list of strings, one per retrieved chunk), and hand it to a Dataset. If you already have a batch evaluation script or a set of logged production queries, this same shape is what you're building toward regardless of how the answers were originally generated.

Running the Ragas evaluation

With the dataset built, running the actual metrics takes a handful of lines.

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

results = evaluate(
    eval_dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
    ],
)

print(results)
df = results.to_pandas()
print(df[["question", "faithfulness", "answer_relevancy", "context_precision", "context_recall"]])

Running this hits your configured LLM multiple times per row — Ragas metrics are themselves LLM-judged, meaning an LLM is asked to break the answer into claims, check each claim against the context, and score the result. This is why evaluation cost and evaluation latency both scale with the number of rows and the number of metrics you choose. For a quick sanity check during development, three to five metrics on a few dozen rows is plenty; for a full regression suite before a release, you can scale to hundreds of rows and let it run as a batch job.

The output df gives you a per-question breakdown, which is where the real value is. Aggregate scores tell you "faithfulness is 0.81 on average," but the per-row table tells you it's actually 0.98 on nine questions and 0.15 on one — and that one row is where you go look at what happened.

Customizing the LLM and embeddings Ragas uses for judging

By default, Ragas metrics use OpenAI models under the hood for judging. If your production pipeline uses a different LLM, or you want the judge model to be different from your generation model (a common and reasonable choice — using a stronger model to judge a cheaper generation model), you configure this explicitly through LangChain wrappers.

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

judge_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o", temperature=0))
judge_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings(model="text-embedding-3-large"))

results = evaluate(
    eval_dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    llm=judge_llm,
    embeddings=judge_embeddings,
)

This is the piece most people miss when they first integrate Ragas: the judge model and the generation model are two independent configuration points. Using a cheap model to generate answers but a stronger model to judge them is a legitimate and common cost-saving pattern — you get high-quality evaluation without paying premium-model prices for every production query.

If you're running an open-source model through something like Ollama or a self-hosted vLLM endpoint, the same LangchainLLMWrapper pattern works as long as you have a LangChain chat model class pointing at it — Ragas doesn't need to know or care what's behind that abstraction.

Generating synthetic test sets with Ragas

Hand-writing question/ground-truth pairs, as we did above, doesn't scale past a handful of examples. Ragas includes a test set generator that reads your document corpus and synthesizes realistic question-answer pairs automatically, including different question complexities (simple, reasoning-based, multi-context).

from ragas.testset import TestsetGenerator
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

generator_llm = ChatOpenAI(model="gpt-4o-mini")
critic_llm = ChatOpenAI(model="gpt-4o")
embeddings = OpenAIEmbeddings()

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings,
)

testset = generator.generate_with_langchain_docs(
    documents=chunks,
    testset_size=20,
)

testset_df = testset.to_pandas()
print(testset_df[["question", "ground_truth", "evolution_type"]].head())

This is genuinely useful once your corpus is larger than a few dozen pages, because it removes the bottleneck of a human writing test questions. The generator uses your actual document chunks — the same chunks object from the LangChain text splitter earlier — as source material, so the synthetic questions are grounded in content that actually exists in your index, which means you can immediately run them through the same evaluation loop from the previous sections.

A word of caution: synthetic test sets are a starting point, not a replacement for real user queries. Real users ask oddly-phrased, incomplete, or multi-part questions that a generator tuned toward "realistic-sounding" text won't always produce. Use synthetic sets to catch broad regressions quickly, and periodically sample real production queries (with PII scrubbed) to keep your evaluation set honest.

Turning this into a repeatable evaluation script

A one-off notebook cell is fine for exploration, but the actual payoff of Ragas comes from running it automatically — after every prompt change, every retriever tweak, every model swap. Here's a script structure that's easy to drop into a CI pipeline or a pre-deploy check.

import sys
import json
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

THRESHOLDS = {
    "faithfulness": 0.75,
    "answer_relevancy": 0.70,
    "context_precision": 0.65,
    "context_recall": 0.70,
}

def run_eval(qa_chain, questions, ground_truths):
    answers, contexts = [], []
    for q in questions:
        result = qa_chain.invoke({"query": q})
        answers.append(result["result"])
        contexts.append([d.page_content for d in result["source_documents"]])

    dataset = Dataset.from_dict({
        "question": questions,
        "answer": answers,
        "contexts": contexts,
        "ground_truth": ground_truths,
    })

    result = evaluate(
        dataset,
        metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    )
    return result.to_pandas()


def check_thresholds(df):
    averages = df[list(THRESHOLDS.keys())].mean().to_dict()
    failures = {
        metric: score for metric, score in averages.items()
        if score < THRESHOLDS[metric]
    }
    return averages, failures


if __name__ == "__main__":
    df = run_eval(qa_chain, questions, ground_truths)
    averages, failures = check_thresholds(df)

    print(json.dumps(averages, indent=2))

    if failures:
        print("Evaluation failed thresholds:", failures)
        sys.exit(1)

    print("All Ragas metrics passed thresholds.")
    sys.exit(0)

Wrap this in a GitHub Actions job (or whatever CI you use) that runs on every pull request touching your prompt templates, retriever config, or chunking logic. A non-zero exit code fails the check, and you catch a regression in retrieval quality or a prompt change that increased hallucination rate before it reaches production — the same way you'd catch a broken unit test.

Common pitfalls when integrating Ragas with LangChain

A few issues come up repeatedly enough to call out directly.

• `contexts` must be a list of strings, not `Document` objects. LangChain retrievers return Document objects with .page_content and .metadata. Ragas wants raw strings. Forgetting the .page_content extraction is the single most common error when people first wire this up, and it fails with a confusing schema error rather than an obvious one.
• Mismatched judge and generation models inflate scores. If you use the same model to generate an answer and to judge its own faithfulness, you can get artificially high scores because the model is consistent with itself, not necessarily correct. Where budget allows, use a stronger or at least different model as the judge.
• Context recall requires a real `ground_truth`. If you skip the ground truth column, you can still compute faithfulness, answer relevancy, and context precision, but context recall will fail or be meaningless — it's fundamentally a comparison against a reference answer.
• Async batching matters at scale. Ragas evaluation calls the LLM per metric per row. At even 100 rows with four metrics, that's potentially 400+ LLM calls. Ragas batches and parallelizes these internally, but if you're hitting rate limits, reduce batch_size or drop to fewer metrics for fast iteration, then run the full set overnight for a release gate.
• Chunking changes invalidate old evaluation baselines. If you change your chunk_size or chunk_overlap, your context precision and recall numbers will shift even if retrieval quality is unchanged in spirit, because the shape of what counts as "a chunk" is different. Re-baseline after chunking changes rather than comparing directly against old scores.

Where to go from here

Ragas turns "does my RAG pipeline work" from a vibe into a number you can track over time, diff across pull requests, and use to justify infrastructure decisions — whether that's switching vector stores, changing chunk sizes, or upgrading your generation model. The integration with LangChain is intentionally low-friction: your chain doesn't need to change at all, you just need to capture question, answer, and contexts on the way out and feed them into evaluate().

Start small. Pick five real questions your users actually ask, run them through the loop in this guide, and look at the per-row breakdown before you look at the averages. The averages will tell you a story that sounds fine. The per-row table will tell you where it isn't.

If you want a structured walkthrough that goes deeper into metric internals, synthetic test set tuning, and building evaluation dashboards on top of Ragas output, our Ragas Tutorial course on teachyou.ai covers the full workflow from first principles through production CI integration, taught alongside the same kind of real, runnable pipelines used in this guide.