Ragas Answer Correctness Metric Explained

Why "Correctness" Is Harder Than It Sounds

Every RAG pipeline eventually asks the same question: is this answer actually right? Not "does it sound right," not "is it fluent," but is it factually aligned with the ground truth you expect. That sounds simple until you try to grade it programmatically. Two answers can use completely different words and still mean the same thing. Two answers can share almost every word and still contradict each other on the one fact that matters.

This is the exact problem Ragas built AnswerCorrectness to solve. Most people evaluating retrieval-augmented generation systems start with simpler metrics — faithfulness, answer relevancy, context precision — and those matter, but none of them tell you whether the generated answer matches the reference answer in substance. AnswerCorrectness is the metric that does, and it does it by combining two very different signals: factual overlap and semantic similarity.

If you've ever shipped a RAG system to production and had a stakeholder ask "but how do we know the answers are correct," this is the metric you reach for. It's the closest thing Ragas has to a single number that captures ground-truth alignment. In this article we'll break down exactly how it works internally, how to compute it with the AnswerCorrectness class, how to interpret and tune the score, and where it fits next to the other metrics in the Ragas suite.

What AnswerCorrectness Actually Measures

AnswerCorrectness in Ragas is a composite metric. Instead of relying on a single heuristic, it evaluates the generated response against a reference answer along two axes:

• Factual correctness — computed via classification of statements into true positives, false positives, and false negatives, then rolled up into an F1 score.
• Semantic similarity — computed via embedding-based cosine similarity between the response and the reference.

The final score is a weighted sum of these two components. By default, Ragas weights factual correctness much more heavily than semantic similarity — the default weights are [0.75, 0.25]. That's a deliberate design choice: two answers can be semantically similar (same topic, same vocabulary) while disagreeing on the actual facts, and Ragas wants the score to reflect that a factually wrong-but-fluent answer is still a bad answer.

Here's the breakdown of the factual side. The metric uses an LLM to decompose both the response and the reference into atomic statements, then classifies each statement into one of three buckets:

• TP (True Positive): statements present in both the response and the reference, i.e., facts that are correctly stated.
• FP (False Positive): statements present in the response but not supported by the reference, i.e., hallucinated or incorrect facts.
• FN (False Negative): statements present in the reference but missing from the response, i.e., facts the answer failed to include.

From these three counts, Ragas computes a standard F1 score:

F1 = TP / (TP + 0.5 * (FP + FN))

This is precision and recall folded into one number. A response that hallucinates extra facts gets penalized through FP. A response that's technically accurate but incomplete gets penalized through FN. A response that nails every fact in the reference, and only those facts, scores close to 1.0 on the factual component.

The semantic similarity component is more straightforward — it embeds both texts (response and reference) using an embedding model and computes cosine similarity between the vectors. This catches cases where the factual decomposition might be too strict about phrasing, giving partial credit for answers that are conceptually aligned even if the atomic-fact-matching missed something.

Setting Up the Environment

Before writing any evaluation code, get your environment in order. AnswerCorrectness needs both an LLM (for statement generation and classification) and an embedding model (for the semantic similarity component), so you need both configured.

pip install ragas langchain-openai datasets

import os

os.environ["OPENAI_API_KEY"] = "sk-your-key-here"

from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))
evaluator_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

You don't have to use OpenAI here — Ragas supports any LangChain or LlamaIndex-compatible model wrapper, so Anthropic, Azure OpenAI, or local models via Ollama all work the same way. What matters is that the LLM you pick is competent at decomposing sentences into discrete factual claims and judging entailment between them — smaller or weaker models tend to produce noisier TP/FP/FN classifications, which shows up as more score variance across repeated runs.

Computing AnswerCorrectness Directly, Then At Scale

The cleanest way to understand the metric is to run it on a single example before wiring it into a full evaluation pipeline. Here's a minimal, single-sample example using the SingleTurnSample object and the AnswerCorrectness class directly:

from ragas import SingleTurnSample
from ragas.metrics import AnswerCorrectness
import asyncio

sample = SingleTurnSample(
    user_input="When was the first Ragas paper published and what does it evaluate?",
    response=(
        "The Ragas framework paper was published in 2023. "
        "It focuses on reference-free evaluation of retrieval-augmented "
        "generation systems using LLMs as judges."
    ),
    reference=(
        "Ragas was introduced in 2023 as a framework for automated, "
        "reference-free evaluation of RAG pipelines, covering metrics "
        "for faithfulness, answer relevancy, and context precision."
    ),
)

answer_correctness = AnswerCorrectness(
    llm=evaluator_llm,
    embeddings=evaluator_embeddings,
    weights=[0.75, 0.25],
)

async def main():
    score = await answer_correctness.single_turn_ascore(sample)
    print(f"Answer Correctness: {score:.4f}")

asyncio.run(main())

Notice the weights parameter. This is the lever you'll reach for most often when tuning the metric — more on that shortly. Also notice that response and reference are both required fields on SingleTurnSample. AnswerCorrectness will raise an error if reference is missing, since without a ground-truth answer there's nothing to compare against. This is one of the key differences from metrics like Faithfulness, which only need retrieved_contexts and response and don't require a reference answer at all.

In practice you're evaluating dozens or hundreds of question-answer pairs, not one. Ragas' evaluate function handles batching, async execution, and result aggregation for you.

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_correctness

data = {
    "user_input": [
        "What is the capital of Australia?",
        "Who wrote 'Pride and Prejudice'?",
        "What is the boiling point of water at sea level?",
    ],
    "response": [
        "Sydney is the capital of Australia.",
        "Jane Austen wrote 'Pride and Prejudice' in 1813.",
        "Water boils at 100 degrees Celsius at sea level.",
    ],
    "reference": [
        "Canberra is the capital of Australia.",
        "Jane Austen wrote 'Pride and Prejudice', published in 1813.",
        "At sea level, water boils at 100 degrees Celsius (212 Fahrenheit).",
    ],
}

dataset = Dataset.from_dict(data)

results = evaluate(
    dataset=dataset,
    metrics=[answer_correctness],
    llm=evaluator_llm,
    embeddings=evaluator_embeddings,
)

df = results.to_pandas()
print(df[["user_input", "response", "answer_correctness"]])

Run this and you'll see something instructive: the Canberra/Sydney example should score noticeably lower than the other two, because the model gives a factually wrong capital city. The embedding similarity component will still pull the score up somewhat — "Sydney is the capital of Australia" and "Canberra is the capital of Australia" are structurally almost identical sentences, so cosine similarity between them is high even though the fact is wrong. This is exactly the scenario the 0.75/0.25 weighting is designed for: without leaning hard on the factual F1 component, a wrong-but-similarly-worded answer could score deceptively well.

The Jane Austen example should score close to 1.0 — the response includes every fact in the reference and adds nothing false. The boiling point example is a good test of the statement decomposition step, since it restates the same fact with a Fahrenheit conversion added; a well-tuned classifier should treat that as consistent rather than penalizing it as an unsupported addition.

Tuning the Weights and Understanding Trade-offs

The default weights=[0.75, 0.25] (factual, semantic) work well as a general default, but you should reconsider them depending on the type of application you're evaluating.

If you're building something like a legal or medical Q&A system, you want factual precision to dominate almost completely — a fluent-sounding answer that gets a dosage or clause wrong is unacceptable regardless of how semantically close it reads. In that case, push the weighting further toward factual correctness:

strict_factual = AnswerCorrectness(
    llm=evaluator_llm,
    embeddings=evaluator_embeddings,
    weights=[0.9, 0.1],
)

If instead you're evaluating a conversational or summarization-style system where paraphrasing and different levels of detail are expected and acceptable, you might weight semantic similarity higher so that reasonable rewordings aren't punished too harshly:

paraphrase_tolerant = AnswerCorrectness(
    llm=evaluator_llm,
    embeddings=evaluator_embeddings,
    weights=[0.5, 0.5],
)

There's no universally correct weighting — this is a design decision you make based on what "correct" means for your product. What matters is that you pick weights deliberately, document them, and keep them consistent across evaluation runs so scores stay comparable over time. If you change the weighting mid-project, treat it like a metric version bump — old scores are no longer directly comparable to new ones.

One subtlety worth flagging: the factual correctness F1 depends entirely on the quality of statement decomposition, which depends entirely on the LLM you use as the judge. A weaker judge model will sometimes over-split statements (turning one fact into three overlapping ones) or under-split them (missing a contradiction embedded inside a longer sentence). If you see unstable scores across repeated runs on the same data, the judge model is usually the first thing to check, not the pipeline you're evaluating.

Comparing AnswerCorrectness to Neighboring Metrics

It's easy to conflate AnswerCorrectness with other Ragas metrics that sound similar. Here's how they differ in practice:

• AnswerCorrectness vs. Faithfulness — Faithfulness checks whether the response is supported by the *retrieved context*, not by a ground-truth reference. A response can be perfectly faithful to a bad or incomplete context and still be factually wrong relative to reality. AnswerCorrectness is the one that checks against ground truth.
• AnswerCorrectness vs. AnswerSimilarity — AnswerSimilarity (sometimes called answer_similarity or SemanticSimilarity depending on the Ragas version) is literally just the embedding-based cosine similarity component of AnswerCorrectness, computed on its own. AnswerCorrectness wraps that plus the factual F1 layer on top.
• AnswerCorrectness vs. ContextPrecision / ContextRecall — those two evaluate the retrieval step, i.e., whether the right chunks were pulled from your vector store. AnswerCorrectness evaluates the generation step's output against a reference, assuming retrieval already happened.

If you're building a full RAG evaluation harness, the natural pattern is to run retrieval metrics (context precision, context recall) alongside generation metrics (faithfulness, answer relevancy) alongside AnswerCorrectness, so you can localize failures. If context recall is low but AnswerCorrectness is also low, the problem is probably upstream — your retriever isn't finding the right passages. If context recall is high but AnswerCorrectness is still low, your generator is the bottleneck — it has the right information and still produces a wrong or incomplete answer.

from ragas.metrics import (
    answer_correctness,
    faithfulness,
    context_precision,
    context_recall,
)

full_suite_results = evaluate(
    dataset=dataset,
    metrics=[context_precision, context_recall, faithfulness, answer_correctness],
    llm=evaluator_llm,
    embeddings=evaluator_embeddings,
)

print(full_suite_results.to_pandas())

Running the full suite side by side like this is how you actually diagnose a RAG pipeline instead of just scoring it. A single AnswerCorrectness number tells you something is wrong; the full suite tells you where.

Common Pitfalls When Using AnswerCorrectness

A few mistakes show up repeatedly when teams first adopt this metric:

• Missing or low-quality reference answers. AnswerCorrectness is only as good as the reference field you provide. If your ground-truth answers are themselves vague, incomplete, or inconsistent in style, the metric will faithfully report noise. Invest time in writing clear, complete reference answers before you trust the scores.
• Treating the score as absolute rather than relative. A single AnswerCorrectness number in isolation ("0.71") doesn't mean much on its own. It becomes useful when you track it across prompt changes, model swaps, or retrieval tweaks — use it as a regression signal, not a pass/fail gate, unless you've calibrated a threshold against your own data.
• Ignoring judge-model cost and latency. Because the metric requires an LLM call to decompose and classify statements (on top of the embedding call), running it over large evaluation sets can get slow and expensive if you're using a large model as the judge. A smaller, cheaper model is often good enough for the judge role — validate this on a sample before committing to it at scale.
• Not fixing the weights before comparing runs. If you tune weights between evaluation runs without tracking it, you'll draw the wrong conclusions about whether your pipeline improved or your scoring criteria changed.
• Forgetting async batching for large datasets. The single_turn_ascore coroutine is meant to be run per-sample; for full datasets, let evaluate() handle concurrency rather than looping synchronously, or you'll pay for it in wall-clock time.

Wiring AnswerCorrectness Into a Test Suite, and Debugging Low Scores

Once you trust the metric on a handful of examples, the next step is usually to turn it into an automated regression check rather than something you run manually in a notebook. A common pattern is to build a small pytest suite that fails a build if AnswerCorrectness drops below an agreed threshold on a fixed evaluation set.

import pytest
import asyncio
from ragas import SingleTurnSample
from ragas.metrics import AnswerCorrectness

THRESHOLD = 0.7

golden_set = [
    {
        "user_input": "What port does the default Ragas embedding server run on?",
        "response": "Ragas does not run its own embedding server; it calls whichever embedding provider you configure, such as OpenAI or a local model.",
        "reference": "Ragas has no built-in embedding server. It delegates embedding calls to the provider you wire in through LangchainEmbeddingsWrapper or LlamaIndexEmbeddingsWrapper.",
    },
    {
        "user_input": "Does AnswerCorrectness require a reference answer?",
        "response": "Yes, AnswerCorrectness requires a reference field, since it compares the response against ground truth.",
        "reference": "Yes. Unlike Faithfulness, AnswerCorrectness needs a reference answer to compute both the factual F1 score and semantic similarity.",
    },
]

@pytest.mark.asyncio
async def test_answer_correctness_regression():
    metric = AnswerCorrectness(llm=evaluator_llm, embeddings=evaluator_embeddings)
    scores = []
    for row in golden_set:
        sample = SingleTurnSample(**row)
        score = await metric.single_turn_ascore(sample)
        scores.append(score)

    average_score = sum(scores) / len(scores)
    assert average_score >= THRESHOLD, (
        f"AnswerCorrectness regressed to {average_score:.3f}, below threshold {THRESHOLD}"
    )

This kind of check plugs neatly into a CI pipeline that runs on every prompt change, retriever swap, or model upgrade. The key design decision is choosing your golden set carefully — it should be small enough to run quickly and cheaply on every commit, but representative enough that a regression in real user-facing quality actually shows up as a drop in this number. Many teams maintain two tiers: a small "smoke test" golden set (10-20 examples) that runs on every pull request, and a larger nightly evaluation set (hundreds of examples) that runs on a schedule and produces trend charts over time.

It's worth being honest about the trade-off here: because AnswerCorrectness makes LLM calls, this test is slower and less deterministic than a typical unit test. Two consecutive runs on identical inputs can differ slightly if your judge model has any temperature above zero, or if the underlying model itself changes behavior between provider-side updates. Set your evaluator LLM's temperature to 0 where possible, and treat the threshold as a band rather than an exact cutoff — a single point of noise shouldn't fail your build, but a sustained drop across several runs should.

When a sample comes back with an unexpectedly low AnswerCorrectness score, resist the urge to immediately blame the metric. Walk through the decomposition manually first. Take the response and reference, and ask yourself: if I were grading this by hand, what atomic facts would I extract from each, and which ones actually conflict?

A frequent cause of surprising low scores is reference answers that bundle in extra, tangential facts the question never asked for. If your reference answer for "What is the boiling point of water at sea level?" also mentions atmospheric pressure in pascals, freezing point, and altitude effects, and your system's response only answers the boiling point question directly, you'll rack up false negatives for facts that were never actually required to answer well. This isn't a bug in AnswerCorrectness — it's doing exactly what it's told — but it does mean your reference answers need to be scoped to what you actually expect a good response to contain, not padded with everything that's true and topically related.

Another frequent cause is a judge model that splits compound sentences inconsistently between the response and reference. If the response says "Canberra is the capital of Australia, and it was chosen as a compromise between Sydney and Melbourne" and the reference only says "Canberra is the capital of Australia," a strict judge might flag the compromise detail as an unsupported false positive, even though it's true and harmless additional context. If you see this pattern repeatedly, consider adjusting your evaluator prompt behavior by using a stronger judge model, or by pre-processing reference answers to be more complete so the false-positive rate settles down.

You can also isolate which component is dragging the score down by computing the two parts separately. Run AnswerSimilarity on its own to check the semantic side, and manually inspect the LLM's statement extraction to check the factual side. If the semantic similarity score is high but the composite AnswerCorrectness score is low, the factual F1 component is where your bug or your data issue lives — go look at the specific facts being flagged as false positives or false negatives instead of trying to fix the pipeline blindly.

Reading the Score in Context

A final practical point: don't chase a "good" AnswerCorrectness number in the abstract. What counts as good depends heavily on your reference-answer style and your weighting. A knowledge-base FAQ bot with short, template-like reference answers might realistically sit around 0.85–0.95 once tuned. A summarization-heavy application with long, multi-fact reference answers might realistically plateau lower, around 0.6–0.75, simply because complete factual overlap on long-form text is a much higher bar.

The right way to use this metric is comparative: baseline your current pipeline, make one change at a time — a new retriever, a new prompt template, a different generation model — and watch how the AnswerCorrectness score moves relative to your own baseline. That delta is the signal. The raw number, compared across teams or projects with different reference-answer conventions, usually isn't.

It's also worth logging the decomposed TP/FP/FN statement lists when you can access them (Ragas exposes intermediate outputs depending on version and tracing setup), because the aggregate F1 hides exactly *which* facts were hallucinated versus omitted. When you're debugging a low-scoring sample, that breakdown is far more actionable than the single float.

Wrapping Up

AnswerCorrectness earns its place as one of the core Ragas metrics because it answers the question stakeholders actually ask: does this match what we expected? By blending a factual F1 score with semantic similarity, it avoids the two failure modes of simpler approaches — rewarding fluent-but-wrong answers, or punishing correct answers that happen to use different words. The weighting system gives you a real lever to match the metric's behavior to your domain's tolerance for paraphrasing versus its need for factual precision.

Get your reference answers right, pick a judge model you trust, fix your weights, and use the score as a longitudinal signal rather than a one-off grade, and AnswerCorrectness becomes one of the most reliable tools you have for knowing whether your RAG system is actually getting better — not just sounding more confident.

If you want to go deeper — building full evaluation harnesses, wiring Ragas into CI pipelines, and combining AnswerCorrectness with the rest of the metric suite on real production RAG systems — that's exactly what we cover hands-on in the Ragas Tutorial course on teachyou.ai.