Ragas

Ragas Integration with LlamaIndex: Evaluating LlamaIndex Pipelines

Ira Menon · May 8, 2026 · 14 min read

Why your LlamaIndex pipeline needs a real evaluation layer

You built a LlamaIndex pipeline. It ingests documents, chunks them, embeds them, stores them in a vector index, and answers questions through a query engine. It looks like it works — you tried five questions, the answers sounded reasonable, and you shipped it.

Then a user asks something slightly out of distribution and the pipeline confidently hallucinates a fact that appears nowhere in the source documents. Or it retrieves the wrong chunk and answers a completely different question than the one asked. "Looks like it works" is not the same as "is evaluated," and the gap between those two states is where most production RAG incidents come from.

This is exactly the problem Ragas was built to solve. Ragas is an open-source evaluation framework purpose-built for retrieval-augmented generation systems. Instead of eyeballing outputs, it gives you numeric metrics — faithfulness, answer relevancy, context precision, context recall — computed by LLM-based judges against your actual retrieved contexts and generated answers. And critically, Ragas ships a first-class integration for LlamaIndex, so you don't have to hand-roll a bridge between your query engine and the evaluation harness.

In this article we'll build a LlamaIndex query engine from scratch, wire it into Ragas using the official integration, run the core RAG metrics against it, and talk through how to read the results and what to do when a metric comes back low. Everything here is runnable — copy the code blocks into a project with ragas, llama-index, and an OpenAI (or compatible) API key set, and you'll have a working evaluation loop by the end.

What Ragas actually measures

Before touching code, it's worth being precise about what each metric captures, because misreading a Ragas score is almost as bad as not measuring at all.

Faithfulness — does the generated answer only contain claims that can be inferred from the retrieved context? A low faithfulness score means your LLM is hallucinating or extrapolating beyond what was actually retrieved. This is the metric that catches "confidently wrong."
Answer relevancy — does the generated answer actually address the question that was asked? A pipeline can be perfectly faithful to its context and still answer the wrong question if retrieval pulled irrelevant chunks.
Context precision — of the chunks that were retrieved, how many were actually relevant and ranked appropriately? This is a retrieval-quality metric, not a generation-quality metric.
Context recall — did retrieval pull back everything needed to answer the question, measured against a reference/ground-truth answer? This requires labeled reference answers and tells you whether your retriever is missing information entirely.

Notice the split: faithfulness and answer relevancy evaluate the generation side of your pipeline, while context precision and context recall evaluate the retrieval side. This distinction matters enormously for debugging. If faithfulness is low but context precision is high, your retriever is doing its job and the problem is prompt engineering or model choice. If context recall is low, no amount of prompt tuning will fix it — you need better chunking, better embeddings, or a bigger top_k.

Setting up the environment

Install the packages you need. Ragas depends on datasets under the hood (it represents evaluation sets as Hugging Face Dataset objects), and the LlamaIndex integration lives in the core ragas package as of recent releases.

pip install ragas llama-index llama-index-embeddings-openai llama-index-llms-openai
export OPENAI_API_KEY="sk-..."

If you're running against a local corpus, put a handful of .txt or .md files in a data/ directory. For this walkthrough, assume you have a small internal knowledge base — say, a handful of documents describing a company's expense reimbursement policy, remote work policy, and onboarding checklist. Small, deliberately narrow corpora are actually ideal for learning evaluation, because you can manually verify whether an answer is correct without needing domain expertise.

Building a baseline LlamaIndex query engine

Here's a standard LlamaIndex pipeline: load documents, build a VectorStoreIndex, and expose a query engine.

from llama_index.core import (
    SimpleDirectoryReader,
    VectorStoreIndex,
    Settings,
)
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# Configure global LlamaIndex settings
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

# Load and index documents
documents = SimpleDirectoryReader("data/").load_data()
index = VectorStoreIndex.from_documents(documents)

# Build a query engine that returns source nodes alongside the answer
query_engine = index.as_query_engine(similarity_top_k=3)

response = query_engine.query("How many days of remote work are employees allowed per week?")
print(response.response)
print("\n--- Retrieved context ---")
for node in response.source_nodes:
    print(node.node.get_content()[:200], "...\n")

This is intentionally minimal. similarity_top_k=3 means the retriever pulls back the three most similar chunks for every query, and the query engine synthesizes an answer from them. If you run this against a real policy document, you'll get a plausible-sounding answer. The question is whether it's actually grounded in the retrieved chunks, and whether those chunks were the right ones to retrieve in the first place. That's exactly what we can't tell just by reading the output — which is why we bring in Ragas.

Building an evaluation dataset

Ragas needs a set of test cases to run metrics against. Each test case needs, at minimum: a user_input (the question), a response (what your pipeline generated), and retrieved_contexts (the chunks your retriever pulled back). For context recall you additionally need a reference — a ground-truth answer you or a domain expert wrote by hand.

Writing your own evaluation questions (rather than reusing questions the system was tuned on) is the single most important step here. A common mistake is evaluating with the same three questions used during development — that tells you nothing about generalization. Aim for 15-30 questions that span easy lookups, questions requiring synthesis across multiple chunks, and edge cases like questions the corpus genuinely can't answer.

eval_questions = [
    "How many days of remote work are employees allowed per week?",
    "What is the maximum reimbursable amount for a client dinner?",
    "Who approves expense reports over $500?",
    "What laptop model is issued to new engineering hires?",
    "How many vacation days do employees accrue in their first year?",
]

# Hand-written reference answers, used for context recall
eval_references = [
    "Employees may work remotely up to 3 days per week with manager approval.",
    "Client dinners are reimbursable up to $150 per person without pre-approval.",
    "Expense reports over $500 require director-level approval.",
    "New engineering hires receive a 14-inch MacBook Pro by default.",
    "Employees accrue 1.25 vacation days per month worked in their first year.",
]

Now run each question through the query engine and collect the response plus retrieved contexts:

results = {
    "user_input": [],
    "response": [],
    "retrieved_contexts": [],
    "reference": [],
}

for question, reference in zip(eval_questions, eval_references):
    response = query_engine.query(question)
    contexts = [node.node.get_content() for node in response.source_nodes]

    results["user_input"].append(question)
    results["response"].append(str(response))
    results["retrieved_contexts"].append(contexts)
    results["reference"].append(reference)

This loop is the bridge between LlamaIndex and Ragas: it's just walking the query engine and capturing the three artifacts Ragas needs — the question, the answer, and the exact chunks that were retrieved for that answer.

Running Ragas metrics against the pipeline

With the data collected, convert it into a Ragas EvaluationDataset and run the core metrics.

from ragas import evaluate, EvaluationDataset
from ragas.metrics import (
    Faithfulness,
    AnswerRelevancy,
    ContextPrecision,
    ContextRecall,
)
from ragas.llms import LlamaIndexLLMWrapper
from ragas.embeddings import LlamaIndexEmbeddingsWrapper
from llama_index.llms.openai import OpenAI as LlamaIndexOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding as LlamaIndexEmbedding

# Wrap the SAME kind of LLM/embeddings LlamaIndex uses so Ragas judges
# with a consistent, known model rather than defaulting silently
evaluator_llm = LlamaIndexLLMWrapper(LlamaIndexOpenAI(model="gpt-4o", temperature=0))
evaluator_embeddings = LlamaIndexEmbeddingsWrapper(
    LlamaIndexEmbedding(model="text-embedding-3-small")
)

dataset = EvaluationDataset.from_dict(results)

metrics = [
    Faithfulness(llm=evaluator_llm),
    AnswerRelevancy(llm=evaluator_llm, embeddings=evaluator_embeddings),
    ContextPrecision(llm=evaluator_llm),
    ContextRecall(llm=evaluator_llm),
]

report = evaluate(dataset=dataset, metrics=metrics)
print(report)

df = report.to_pandas()
print(df[["user_input", "faithfulness", "answer_relevancy", "context_precision", "context_recall"]])

Two design choices are worth calling out here. First, using LlamaIndexLLMWrapper and LlamaIndexEmbeddingsWrapper means the LLM doing the judging is expressed through LlamaIndex's own abstractions — useful if your project already standardizes on LlamaIndex's LLM interface and you want one less dependency surface to manage. Second, it's deliberate to use a stronger model (gpt-4o) as the judge than the model powering the pipeline itself (gpt-4o-mini). Using a weaker model to judge a stronger model's outputs introduces noise into your metrics — the judge needs enough capability to reliably detect subtle hallucinations.

Running evaluate() will make a batch of LLM calls per metric per row — for five questions and four metrics, expect on the order of a few dozen API calls. This is why real evaluation suites should be run asynchronously and cached; Ragas handles concurrency internally, but for larger datasets you should set a rate limit via RunConfig to avoid tripping API rate limits.

from ragas.run_config import RunConfig

report = evaluate(
    dataset=dataset,
    metrics=metrics,
    run_config=RunConfig(max_workers=4, timeout=60),
)

Reading the scorecard and diagnosing failures

Once you have a dataframe of per-question scores, the real work begins: interpreting them. All four core metrics are scaled 0 to 1, with 1 being best.

Faithfulness near 1.0, answer relevancy low — the pipeline is truthfully summarizing whatever it retrieved, but that content doesn't address the question. This is a retrieval problem masquerading as a generation problem. Check context_precision next.
Faithfulness low, context precision high — the retriever did its job, but the LLM is adding claims not present in the context. This points to the generation prompt. Try tightening the system prompt to explicitly instruct "only answer using the provided context" and consider lowering temperature further or switching models.
Context recall low — the ground-truth answer requires information that was never retrieved at all. No prompt engineering fixes this. You need to either increase similarity_top_k, improve chunking (maybe your reimbursement policy paragraph got split across two chunks and only one made it into top-3), or improve the embedding model.
Everything low on one specific question — look at that question individually. Sometimes it's a genuinely ambiguous question, sometimes the corpus simply doesn't contain the answer, in which case your reference answer might be wrong, not the pipeline.

A pattern worth watching for in real projects: teams tune their pipeline against a small set of "golden" questions until every metric looks great, then discover in production that questions phrased differently than the golden set score much worse. Treat your Ragas evaluation set the way you'd treat a test set in any ML workflow — it should be representative and it should grow over time as you discover new failure modes in production, not shrink to whatever makes the numbers look good.

Comparing two retrieval configurations

The real value of wiring Ragas into LlamaIndex isn't a one-off score — it's the ability to compare configurations objectively. Suppose you want to know whether increasing similarity_top_k from 3 to 5 actually helps.

def build_query_engine(top_k: int):
    return index.as_query_engine(similarity_top_k=top_k)

def run_eval(top_k: int):
    engine = build_query_engine(top_k)
    rows = {"user_input": [], "response": [], "retrieved_contexts": [], "reference": []}
    for question, reference in zip(eval_questions, eval_references):
        response = engine.query(question)
        rows["user_input"].append(question)
        rows["response"].append(str(response))
        rows["retrieved_contexts"].append([n.node.get_content() for n in response.source_nodes])
        rows["reference"].append(reference)

    ds = EvaluationDataset.from_dict(rows)
    return evaluate(dataset=ds, metrics=metrics)

report_top3 = run_eval(top_k=3)
report_top5 = run_eval(top_k=5)

print("top_k=3:", report_top3.to_pandas()[["context_precision", "context_recall"]].mean())
print("top_k=5:", report_top5.to_pandas()[["context_precision", "context_recall"]].mean())

This is the pattern that turns Ragas from a one-time sanity check into a real experimentation harness: hold the evaluation set fixed, vary one pipeline parameter at a time — top_k, chunk size, embedding model, reranker on or off — and let the metrics tell you which change actually moved the needle rather than trusting a gut feeling from reading a handful of outputs.

You can apply the exact same pattern to chunk size:

from llama_index.core.node_parser import SentenceSplitter
from llama_index.core import VectorStoreIndex

def build_index_with_chunking(chunk_size: int, chunk_overlap: int):
    splitter = SentenceSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
    nodes = splitter.get_nodes_from_documents(documents)
    return VectorStoreIndex(nodes)

Swap this into build_query_engine and rerun the same evaluation loop, and you now have a repeatable, numeric answer to "should our chunks be 256 tokens or 512 tokens" instead of a guess.

Handling multi-hop and synthesis questions

Simple lookup questions ("what is X") are the easiest case for both retrieval and evaluation. Real users ask harder questions that require combining information from multiple chunks — "if I work remotely and also travel for a client dinner in the same week, which policies apply?" These multi-hop questions are exactly where context recall tends to reveal weaknesses, because a top_k tuned for single-fact lookups often isn't large enough to pull in every relevant chunk when multiple policies intersect.

When you see this pattern in your Ragas results, a few LlamaIndex-side fixes are worth trying before increasing top_k blindly, since a higher top_k also increases the chance of injecting irrelevant context and hurting faithfulness:

from llama_index.core.postprocessor import SentenceTransformerRerank

# Retrieve a wider net, then rerank down to the best few
reranker = SentenceTransformerRerank(
    model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_n=4
)

query_engine = index.as_query_engine(
    similarity_top_k=10,
    node_postprocessors=[reranker],
)

Retrieving 10 candidates and reranking down to the top 4 with a cross-encoder tends to improve both context precision and context recall simultaneously compared to just raising similarity_top_k on the base retriever, because the reranker has more candidates to choose from but still filters aggressively before they reach the LLM. Re-running your Ragas evaluation after adding a reranker is the only reliable way to confirm this actually happened for your specific corpus and question set — don't assume it worked, measure it.

Automating evaluation in CI

Once you trust your evaluation set, the natural next step is running it automatically whenever the pipeline changes — a new chunking strategy, an embedding model upgrade, a prompt tweak — rather than manually rerunning a notebook. A simple pattern is a script that fails the build if any metric average drops below a threshold.

import sys

MIN_FAITHFULNESS = 0.85
MIN_CONTEXT_RECALL = 0.75

report = evaluate(dataset=dataset, metrics=metrics)
df = report.to_pandas()

avg_faithfulness = df["faithfulness"].mean()
avg_context_recall = df["context_recall"].mean()

print(f"Faithfulness: {avg_faithfulness:.3f}")
print(f"Context recall: {avg_context_recall:.3f}")

if avg_faithfulness < MIN_FAITHFULNESS or avg_context_recall < MIN_CONTEXT_RECALL:
    print("Evaluation gate FAILED — blocking merge.")
    sys.exit(1)

print("Evaluation gate passed.")

Wire this into a CI job that runs on pull requests touching the RAG pipeline, and you've converted "did this change make things worse" from a question someone has to remember to ask into a gate that runs automatically. The thresholds themselves should come from your own baseline — run the eval on your current production pipeline first, record the numbers, and set thresholds slightly below that baseline so normal variance doesn't cause false failures.

Common pitfalls when integrating Ragas with LlamaIndex

A few mistakes come up repeatedly when teams first wire these two together, and knowing them ahead of time will save debugging time.

Forgetting to capture `retrieved_contexts` at the same time as the response. If you query the engine once for the answer and then run a separate retrieval call later to get contexts, you risk a race condition where the underlying index has changed or the retrieval isn't deterministic, producing contexts that don't actually match the answer you're scoring.
Using the same model for pipeline generation and Ragas judging without realizing it. This isn't inherently wrong, but it can mask systematic biases — if the model has a particular blind spot, it may not "notice" that blind spot when judging its own output either.
Evaluating on too few examples and over-trusting the average. Five questions is enough to prove the wiring works, as shown here, but not enough to trust a faithfulness average of 0.92 as representative. Budget for at least 20-30 well-chosen questions before making a real go/no-go decision on a pipeline change.
Ignoring per-question variance. An average score can hide a bimodal distribution — most questions scoring near 1.0 and a handful scoring near 0, which is a very different problem (specific failure mode) than everything scoring a uniform 0.7 (systemic weakness).
Not versioning the evaluation set. As your corpus grows, old questions may become stale or their reference answers may need updating. Keep the eval set under version control alongside the pipeline code so you can track how both evolve together.

Wrapping up

Ragas and LlamaIndex solve two different halves of the same problem: LlamaIndex gets a retrieval-augmented pipeline running, and Ragas tells you, with numbers instead of vibes, whether that pipeline is actually faithful, relevant, and grounded. The integration pattern is straightforward once you've done it once — run your query engine, capture the question, answer, and retrieved contexts, hand them to Ragas as an EvaluationDataset, and read the four core metrics as a diagnostic split between retrieval quality and generation quality.

The real payoff isn't the first evaluation run, though — it's turning this into a repeatable habit: a fixed evaluation set, metrics tracked across every pipeline change, and a CI gate that catches regressions before they reach production. That habit is what separates teams that ship RAG systems they can trust from teams that ship RAG systems they hope work.

If you want to go deeper — building larger synthetic evaluation datasets, tuning custom metrics, and setting up full evaluation pipelines for production RAG systems — our Ragas Tutorial course on teachyou.ai walks through all of it hands-on, from your first evaluate() call to a production-grade CI gate.