Common Ragas Setup Errors and How to Resolve Them

Why Ragas setup breaks more often than it should

If you have spent an afternoon staring at a ValidationError or a wall of openai.APIError tracebacks while trying to run your first Ragas evaluation, you are not alone. Ragas is one of the most useful libraries for evaluating RAG (retrieval-augmented generation) pipelines, but its setup sits at the intersection of three things that each fail in their own creative ways: LLM provider SDKs, async Python, and pydantic schema validation. When all three are wiring together in a single evaluate() call, the error messages you get back are often several layers removed from the actual root cause.

This is the article we wish existed the first time we set up Ragas for a client evaluation project. Below are the errors we have actually hit (and seen students hit) while wiring Ragas into real pipelines, what causes each one, and the exact fix. We are not going to fabricate benchmark numbers or pretend every error has a one-line fix — some of these require you to understand what Ragas is doing under the hood before the fix will make sense. By the end, you should be able to read a Ragas traceback and know within seconds which of these categories it falls into.

Error 1: `ImportError` and dependency version conflicts on install

The very first error most people hit happens before they even write evaluation code — it happens on pip install ragas. Ragas depends on langchain, langchain-core, openai, datasets, and pydantic, and each of those has its own release cadence. A common failure looks like this:

ImportError: cannot import name 'RunnableConfig' from 'langchain_core.runnables'

or

ImportError: cannot import name 'BaseCallbackHandler' from 'langchain.callbacks.base'

Both of these mean your installed langchain / langchain-core version predates (or postdates) the API surface that your installed Ragas version expects. Ragas moves fast, and its pinned dependency ranges in pyproject.toml change between minor versions. If you installed Ragas into an environment that already had an older LangChain-based project in it, pip's dependency resolver often "successfully" installs a combination that is not actually compatible at import time — pip only checks declared version constraints, not actual API compatibility.

The fix that works almost every time: start from a clean virtual environment and let Ragas pull its own dependency tree instead of installing it into an existing LangChain project's environment.

python -m venv ragas-env
source ragas-env/bin/activate
pip install --upgrade pip
pip install ragas

If you specifically need Ragas alongside an existing LangChain app, pin explicitly instead of hoping the resolver figures it out:

pip install "ragas==0.2.10" "langchain-core>=0.3.15,<0.4" "langchain>=0.3.7,<0.4"

Check what's actually installed with pip show ragas langchain langchain-core | grep -E "Name|Version" before you file a bug report — nine times out of ten the fix is a clean environment, not a Ragas bug.

There is a second variant of this error that shows up specifically on Apple Silicon Macs and in some Docker base images, tied to pyarrow (a transitive dependency of datasets, which Ragas uses internally to represent evaluation data):

ImportError: dlopen(/.../pyarrow/lib.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_ARROW_PYARROW_API'

This is a binary compatibility issue, not a version-range issue, and pip install --upgrade pyarrow alone will not fix it if you have mixed conda and pip installs in the same environment. Remove pyarrow entirely and let pip reinstall it fresh, ideally in an environment that has not touched conda:

pip uninstall -y pyarrow datasets
pip install --no-cache-dir datasets

If the error persists, confirm you are running a native arm64 Python interpreter and not an x86_64 build under Rosetta — python3 -c "import platform; print(platform.machine())" should print arm64. An x86 interpreter on an M-series Mac is a surprisingly common cause of obscure dlopen failures, and no amount of pip reinstalling fixes an architecture mismatch.

Error 2: `openai.AuthenticationError` and silent API key issues

The second most common error is an authentication failure, and it comes in a few disguises. The most direct version:

openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-abc***xyz.', 'type': 'invalid_request_error'}}

This one is self-explanatory once you see it, but a more confusing variant shows up when the key is valid but simply not being read:

openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

This happens constantly in notebook environments because people set OPENAI_API_KEY in a .env file but never actually load it. Ragas' default LLM and embeddings wrappers read from the environment at call time — they do not read your .env file for you.

import os
from dotenv import load_dotenv

load_dotenv()  # this line is the one everyone forgets
assert os.environ.get("OPENAI_API_KEY"), "OPENAI_API_KEY not loaded"

A second, sneakier version of this error shows up when you are using Azure OpenAI or a self-hosted model gateway. Ragas' default metrics assume a standard OpenAI-compatible client unless you explicitly wire a custom LLM wrapper. If you just set AZURE_OPENAI_API_KEY and expect Ragas to pick it up automatically, you will get the generic 401 above because Ragas is still trying to hit api.openai.com. The fix is to explicitly construct the LLM wrapper and pass it into every metric:

from langchain_openai import AzureChatOpenAI
from ragas.llms import LangchainLLMWrapper
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy

azure_llm = AzureChatOpenAI(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
    api_version="2024-08-01-preview",
    azure_deployment="gpt-4o-eval",
)
ragas_llm = LangchainLLMWrapper(azure_llm)

result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy],
    llm=ragas_llm,
)

Skipping the explicit llm= argument is the single most common cause of "it works for my teammate but not for me" bug reports in Ragas evaluation code.

There is a third variant worth calling out separately because it looks identical to a bad key but is not: openai.PermissionDeniedError with a message about the organization not being verified for a specific model. Newer reasoning-tier models are sometimes gated behind organization verification, and Ragas will surface this exactly like any other 401/403 from the underlying client:

openai.PermissionDeniedError: Error code: 403 - {'error': {'message': 'Your organization must be verified to use the model `o1`.', 'type': 'invalid_request_error', 'code': 'unsupported_value'}}

If you hit this, the fix has nothing to do with Ragas — verify the organization in your provider dashboard, or drop back to a model your account already has access to (gpt-4o or gpt-4o-mini are safe defaults for evaluation judges and rarely require extra verification).

Error 3: `RateLimitError` and evaluations that die halfway through

Once authentication is sorted, the next wall people hit is rate limiting, especially when evaluating a dataset with more than a handful of rows. Each Ragas metric issues its own LLM call per row, so a dataset of 200 rows evaluated against 4 metrics can mean 800+ API calls fired in a short window. The error looks like this:

openai.RateLimitError: Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-4o-mini in organization org-xxxx on requests per min (RPM): Limit 500, Used 500, Requested 1.', 'type': 'requests', 'param': None, 'code': 'rate_limit_exceeded'}}

Ragas does have built-in retry logic via its RunConfig, but the defaults are tuned for moderate-scale evaluation, not for firing hundreds of concurrent requests at a tier-1 OpenAI account. The fix is to reduce concurrency and let Ragas' backoff handle transient throttling, rather than turning off retries.

from ragas.run_config import RunConfig
from ragas import evaluate

run_config = RunConfig(
    timeout=180,
    max_retries=10,
    max_wait=90,
    max_workers=4,  # default is 16 — this is almost always too aggressive for free/tier-1 accounts
)

result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy],
    run_config=run_config,
)

If you are still hitting 429s at max_workers=4, the actual fix is not a code change — it is requesting a rate limit increase from your provider, or batching your evaluation into smaller chunks with a time.sleep() between batches:

import time

chunks = [dataset.select(range(i, min(i + 50, len(dataset)))) for i in range(0, len(dataset), 50)]
all_results = []
for chunk in chunks:
    r = evaluate(chunk, metrics=[faithfulness, answer_relevancy], run_config=run_config)
    all_results.append(r)
    time.sleep(20)

One more twist: token-per-minute (TPM) limits and requests-per-minute (RPM) limits are enforced separately, and RunConfig only controls request concurrency, not payload size. If retrieved_contexts holds long passages pulled straight from a vector store, you can hit a TPM-based 429 even at max_workers=1, because each request is too large relative to your per-minute token budget. The error body says tokens instead of requests in the type field — check that before assuming concurrency is the problem. The fix here is trimming context length, not reducing worker count:

MAX_CONTEXT_CHARS = 4000

def truncate_contexts(example):
    example["retrieved_contexts"] = [c[:MAX_CONTEXT_CHARS] for c in example["retrieved_contexts"]]
    return example

dataset = dataset.map(truncate_contexts)

Error 4: schema and column-name mismatches

This is the error category that confuses newcomers the most, because the traceback often points at pydantic internals instead of at your dataframe. A typical message:

ValueError: Column 'contexts' is not present in dataset. Please check the column names.

or, in more recent Ragas versions using the SingleTurnSample schema:

pydantic_core._pydantic_core.ValidationError: 1 validation error for SingleTurnSample
retrieved_contexts
  Field required [type=missing, input_value={...}, input_type=dict]

Ragas metrics each declare what columns they require. faithfulness and answer_relevancy need question (or user_input in newer versions) and answer/response; context_precision and context_recall additionally need contexts/retrieved_contexts and, for context_recall, a ground_truth/reference column. The most common mistake is carrying over column names from an older Ragas tutorial (question, answer, contexts, ground_truths) into a newer Ragas version that renamed these to user_input, response, retrieved_contexts, and reference.

Check your installed version first, since this determines which schema applies:

python -c "import ragas; print(ragas.__version__)"

For Ragas 0.2.x and later, build your dataset with the new field names:

from datasets import Dataset

data = {
    "user_input": ["What is the refund window for annual plans?"],
    "response": ["Annual plans can be refunded within 30 days of purchase."],
    "retrieved_contexts": [[
        "Refund policy: Annual subscriptions are refundable within 30 days.",
        "Monthly subscriptions are non-refundable after activation.",
    ]],
    "reference": ["Annual plans are refundable within 30 days of purchase."],
}
eval_dataset = Dataset.from_dict(data)

If you are following an older tutorial or an older codebase that still uses question/answer/contexts/ground_truths, either pin Ragas to a matching version (pip install "ragas==0.1.21") or rename your columns explicitly before calling evaluate(). Do not guess — run dataset.column_names and cross-reference it against the metric's required fields, which you can inspect directly:

from ragas.metrics import context_recall
print(context_recall.required_columns)

Error 5: `asyncio` event loop errors in notebooks

Ragas runs its metric computations asynchronously under the hood for concurrency, which is great for throughput but causes a very specific class of error inside Jupyter and Colab notebooks:

RuntimeError: asyncio.run() cannot be called from a running event loop

This happens because Jupyter kernels already run their own event loop, and Ragas' internal asyncio.run() calls collide with it. You will not see this error running the same script from a plain terminal, which is what makes it confusing — it is purely a notebook-environment problem.

The standard fix is nest_asyncio, which patches the event loop to allow nested run() calls:

import nest_asyncio
nest_asyncio.apply()

from ragas import evaluate
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy])

Apply nest_asyncio.apply() at the very top of the notebook, before importing Ragas, not right before the evaluate() call — we have seen cases where import order still matters depending on what else has already touched the event loop (LangChain callbacks, for instance, sometimes grab it first). If you are running Ragas inside an already-async framework (a FastAPI endpoint, for example), skip nest_asyncio entirely and use Ragas' native async entrypoint instead:

import asyncio
from ragas import aevaluate  # available in recent Ragas versions

async def run_eval():
    return await aevaluate(dataset, metrics=[faithfulness, answer_relevancy])

result = asyncio.run(run_eval())

Error 6: embeddings misconfiguration for context-based metrics

Metrics like context_precision, context_recall, and answer_similarity need an embeddings model in addition to an LLM, and this dependency is easy to miss because the error surfaces late:

ValueError: Embeddings not set. Please set the embeddings using the `embeddings` parameter.

or, if you did pass embeddings but from an incompatible provider:

openai.NotFoundError: Error code: 404 - {'error': {'message': 'The model `text-embedding-ada-002` does not exist or you do not have access to it', 'type': 'invalid_request_error', 'code': 'model_not_found'}}

The second one is common when teams use a proxy or self-hosted gateway (like a local vLLM server or an Azure deployment) that does not have the exact embedding model name Ragas defaults to. Just as with the LLM wrapper, you need to construct and pass embeddings explicitly rather than relying on defaults:

from langchain_openai import OpenAIEmbeddings
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.metrics import context_precision, context_recall

embeddings = LangchainEmbeddingsWrapper(
    OpenAIEmbeddings(model="text-embedding-3-small")
)

result = evaluate(
    dataset,
    metrics=[context_precision, context_recall],
    llm=ragas_llm,
    embeddings=embeddings,
)

If you are running fully local with something like Ollama or a local embedding server, use the matching wrapper (OllamaEmbeddings, or a custom BaseRagasEmbeddings subclass) rather than assuming evaluate() will fall back gracefully — it will not, and the failure mode is exactly the 404 above, just with your local model name instead.

A related but distinct error shows up when the embeddings call succeeds but returns a dimension mismatch, usually because a metric config was cached from a previous run with a different embedding model:

ValueError: shapes (1,1536) and (1,3072) not aligned: 1536 (dim 1) != 1 (dim 0)

text-embedding-3-small returns 1536-dimensional vectors while text-embedding-3-large returns 3072-dimensional vectors, and switching models mid-project without clearing cached embeddings or similarity precomputations produces a shape mismatch instead of a clear "model changed" warning. Make sure every embeddings-dependent artifact — cached vectors, precomputed similarity matrices, saved Dataset objects with an embeddings column baked in — was generated with the exact model you are currently passing to evaluate(). When in doubt, regenerate rather than reuse.

Error 7: `nan` scores and silently failing metric rows

This one does not raise an exception at all, which makes it more dangerous than the errors above — Ragas will happily hand you back a results object where some rows show nan for a given metric instead of a numeric score. If you are not checking for this, you will report an average score that silently excludes failed rows, quietly overstating your pipeline's quality.

result = evaluate(dataset, metrics=[faithfulness, context_recall])
df = result.to_pandas()
print(df[df.isna().any(axis=1)])

nan rows are almost always caused by one of two things: the LLM judge returned output that Ragas' parser could not extract a verdict from (common with smaller or heavily quantized local models that do not reliably follow the structured-output prompt Ragas uses internally), or a single row in the batch hit a transient API error that got swallowed by Ragas' per-row exception handling rather than propagated. Turn on verbose logging to see which:

import logging
logging.getLogger("ragas").setLevel(logging.DEBUG)

If the cause is a weaker judge model failing to follow Ragas' structured prompt format, switch the evaluator LLM to a stronger model for the judging step specifically — it does not have to be the same model your pipeline uses in production. It is completely normal, and often cheaper, to use a lightweight model in your RAG pipeline while using gpt-4o or an equivalent as the Ragas judge, since evaluation quality depends heavily on the judge's instruction-following ability.

Error 8: testset generation failures with `TestsetGenerator`

Everything above assumes you already have an evaluation dataset with questions, answers, and contexts. But a large share of Ragas users start earlier than that, using TestsetGenerator to synthetically generate question-answer pairs from a corpus of documents. This step has its own error surface, and it fails differently from evaluate().

The most common failure here is a KeyError or ValueError thrown deep inside the knowledge graph construction step, usually because the input documents are too short or too repetitive for Ragas to extract meaningful relationships between them:

ValueError: No clusters were found in the given documents. Try increasing chunk size or adding more documents.

Ragas' testset generation builds a knowledge graph from your documents, extracts entities and relationships, and then synthesizes questions from that graph. If you feed it three short FAQ paragraphs, there simply is not enough structure for it to build a meaningful graph. The fix is usually one of two things: increase the size and diversity of the source corpus (a few dozen substantial documents work far better than a handful of short ones), or explicitly configure the chunking used during graph construction:

from ragas.testset import TestsetGenerator
from langchain_community.document_loaders import DirectoryLoader

loader = DirectoryLoader("./docs", glob="**/*.md")
documents = loader.load()

generator = TestsetGenerator(llm=ragas_llm, embedding_model=embeddings)
testset = generator.generate_with_langchain_docs(
    documents,
    testset_size=20,
)

If generation succeeds but produces obviously low-quality or repetitive questions, that is usually a corpus diversity problem rather than a configuration bug — Ragas can only synthesize questions from the relationships actually present in your documents, so a corpus that repeats the same three facts in different words will produce a testset that does the same.

A second common failure in this step is a plain timeout, since knowledge graph construction issues many more LLM calls per document than a single evaluate() row does:

TimeoutError: Request timed out after 60 seconds.

The fix is the same RunConfig lever as before, but applied to the generator rather than to evaluate():

from ragas.run_config import RunConfig

generator = TestsetGenerator(
    llm=ragas_llm,
    embedding_model=embeddings,
    run_config=RunConfig(timeout=300, max_retries=5),
)

Budget more wall-clock time for testset generation than you would for evaluation itself — generating 20 high-quality synthetic questions from a real document set can take several minutes even with a fast judge model, and cutting the timeout short mid-graph-construction produces the ValueError above rather than a clean retry.

A pre-flight checklist before you run `evaluate()`

Given everything above, here is the sequence we now walk through before every Ragas run, in order:

1. Confirm the Ragas version and cross-check it against the tutorial or docs you are following: python -c "import ragas; print(ragas.__version__)".
2. Confirm your dataset's column names match what your chosen metrics require via metric.required_columns.
3. Confirm OPENAI_API_KEY (or your provider's equivalent) is actually loaded in the current process, not just present in a .env file on disk.
4. Explicitly construct and pass both llm= and embeddings= rather than relying on defaults, especially outside of plain OpenAI.
5. Set a conservative RunConfig(max_workers=...) before evaluating anything beyond a handful of rows.
6. If in a notebook, call nest_asyncio.apply() before importing Ragas.
7. After every run, check result.to_pandas() for nan rows before trusting the aggregate score.

Most Ragas setup pain comes from steps 3 and 4 — the library assumes you know it needs an explicit LLM and embeddings wrapper, and it fails in unhelpful ways when you don't provide them. Once you internalize that Ragas metrics are just LLM-and-embeddings-powered judges wrapped in a pydantic schema, most of these errors stop being mysterious and start being obvious at a glance.

Where to go deeper

Debugging these errors in isolation teaches you the symptoms, but understanding why Ragas is built this way — why it separates the LLM judge from your pipeline's model, why it validates schemas this strictly, why async is baked in at the core — is what actually makes you fast at fixing the next error you have not seen yet. That is exactly the gap our Ragas Tutorial course on teachyou.ai is built to close: we walk through building a full evaluation pipeline from scratch, wiring custom LLM and embeddings providers, structuring datasets correctly for every metric, and setting up evaluations that scale past a handful of rows without falling over. If you are past the "copy-paste from the quickstart" stage and want to actually understand the tool you are debugging, that is where to start.