LLM Eval

Evaluating Retrieval-Free vs RAG Answers Side by Side

Ira Menon · Jun 6, 2026 · 14 min read

Why "just add RAG" is not a strategy

Every team building an LLM product eventually hits the same fork in the road. A stakeholder asks, "should we add retrieval so the model answers from our docs?" and the instinctive answer is yes, obviously, hallucinations are bad, ground it in real data. Six weeks later there's a vector database, a chunking pipeline, a reranker, and an answer quality that is... about the same as before, except now it's slower and costs more per query.

This happens constantly because teams skip the step that actually matters: measuring whether retrieval improves answers *for their specific task* before committing to the infrastructure. Retrieval-Augmented Generation (RAG) is not free. It adds latency, a whole new failure surface (bad chunking, stale indexes, retriever misses), and engineering overhead. It only earns its keep when it measurably beats a retrieval-free baseline — the same model answering from parametric knowledge and whatever context you stuff directly into the prompt.

The only way to know is to run both systems against the same questions and score the outputs side by side, with a rubric that's honest about what "better" means for your use case. This article walks through how to build that evaluation: what to compare, how to structure the harness, which metrics catch the failure modes that matter, and how to use an LLM-as-a-judge to scale the comparison without drowning in manual review. This is a companion topic to prompt evaluation and agent evaluation, both covered elsewhere on this site — if you haven't set up a basic eval harness yet, start there first.

What "retrieval-free" actually means here

Retrieval-free doesn't mean "no context at all." It means the model isn't dynamically querying an external knowledge store at inference time. There are three flavors worth distinguishing, because conflating them produces misleading comparisons:

Pure parametric. The model answers using only what it learned during training. No external text touches the prompt. This is your true baseline — it tells you what the model already knows.
Long-context stuffing. You paste the entire relevant document (or several) directly into the prompt, no retrieval step, no chunking, no similarity search. The model sees everything and has to find the needle itself.
Static few-shot context. A fixed, hand-picked set of examples or reference snippets baked into the system prompt, the same for every query regardless of what's asked.

RAG systems, by contrast, run a retrieval step per query: embed the question, search a vector index (or BM25, or a hybrid), pull back the top-k chunks, and stuff those into the prompt. The comparison you want to run is RAG against whichever retrieval-free variant is the honest alternative for your product. If your corpus is 40 pages, long-context stuffing is a fair fight and RAG might lose. If your corpus is 40,000 pages, long-context stuffing isn't a real option and the fair fight is parametric-only versus RAG.

Getting this baseline right is half the eval. Teams that compare RAG only against pure parametric knowledge are stacking the deck — of course grounding in real documents beats a model guessing from memory on a question about your internal API. The real question is whether retrieval beats the cheaper alternatives available to you.

Designing the comparison set

You need a question set that actually exercises the differences between the two systems, not one that's uniformly easy or uniformly impossible for both. A good comparison set has four buckets:

In-distribution factual lookups. Questions with a clear, verifiable answer that exists in the corpus. "What is the cancellation policy for annual plans?" This is where RAG should win if it's working — the answer is a fact the model can't know unless it's grounded.
Questions the model already knows. General knowledge or well-documented public facts that overlap with training data. "What HTTP status code means 'not found'?" Here retrieval-free should hold its own or win, since retrieval adds latency for zero benefit.
Multi-hop or synthesis questions. Answers that require combining facts from two or more places. "Which pricing tier includes both SSO and the audit log feature?" This is where retrieval quality (or its absence) shows up starkly — bad chunking splits the two facts into different chunks and the retriever may only fetch one.
Out-of-corpus or trick questions. Questions with no answer in the documents at all. "What's our refund policy for a product we don't sell?" This tests whether the system hallucinates versus correctly says "I don't have that information."

Aim for at least 15-20 questions per bucket, sourced from real user queries if you have logs, or written by someone who knows the domain if you don't. Resist the urge to write questions that are all trivially easy — that's how teams end up shipping RAG systems that look great in a demo and fall apart on real traffic.

Building the side-by-side harness

The harness itself is mechanically simple: run each question through both systems, capture the full trace, and store results in a format that supports later scoring. Here's a minimal structure in Python:

import json
from dataclasses import dataclass, asdict
from typing import Optional

@dataclass
class EvalResult:
    question_id: str
    question: str
    bucket: str  # "factual", "known", "multihop", "out_of_corpus"
    system: str  # "retrieval_free" or "rag"
    answer: str
    retrieved_chunks: Optional[list] = None
    latency_ms: float = 0.0
    reference_answer: Optional[str] = None

def run_retrieval_free(question: str, model_client) -> EvalResult:
    response = model_client.generate(
        system_prompt="Answer from your own knowledge only.",
        user_prompt=question,
    )
    return EvalResult(
        question_id=hash(question),
        question=question,
        bucket="",  # filled in by caller
        system="retrieval_free",
        answer=response.text,
        latency_ms=response.latency_ms,
    )

def run_rag(question: str, retriever, model_client, top_k: int = 5) -> EvalResult:
    chunks = retriever.search(question, top_k=top_k)
    context = "\n\n".join(c.text for c in chunks)
    response = model_client.generate(
        system_prompt=(
            "Answer the question using only the provided context. "
            "If the context doesn't contain the answer, say so explicitly."
        ),
        user_prompt=f"Context:\n{context}\n\nQuestion: {question}",
    )
    return EvalResult(
        question_id=hash(question),
        question=question,
        bucket="",
        system="rag",
        answer=response.text,
        retrieved_chunks=[c.text for c in chunks],
        latency_ms=response.latency_ms,
    )

def run_comparison(questions: list, retriever, model_client) -> list:
    results = []
    for q in questions:
        rf = run_retrieval_free(q["text"], model_client)
        rag = run_rag(q["text"], retriever, model_client)
        for r in (rf, rag):
            r.bucket = q["bucket"]
            r.reference_answer = q.get("reference_answer")
        results.extend([rf, rag])
    return results

A few details in this harness matter more than they look. First, capture retrieved_chunks even though you're comparing final answers — when RAG loses, you need to know whether it lost because retrieval failed to find the right chunk, or because the model failed to use a chunk it was given. Those are different bugs with different fixes. Second, capture latency for every call; RAG's cost is not just infrastructure, it's the extra round-trip, and your eval report should surface that trade-off explicitly rather than burying it. Third, the RAG system prompt above explicitly instructs the model to say when context doesn't contain the answer — without that instruction, RAG systems tend to hallucinate answers from parametric knowledge even when retrieval came back empty, which defeats the entire purpose of adding retrieval in the first place.

Metrics that actually distinguish the two systems

Generic "is this answer good" scoring will not tell you what you need to know. You want metrics that isolate *why* one system beat the other.

Correctness against a reference answer. Binary or graded match against a known-good answer, where one exists. This is your ground truth signal and should carry the most weight for the factual and multi-hop buckets.
Faithfulness (groundedness). For RAG specifically: does every claim in the answer trace back to something in the retrieved chunks? This catches the case where retrieval succeeded but generation ignored it and answered from parametric memory anyway.
Abstention accuracy. For the out-of-corpus bucket: did the system correctly decline to answer, or did it confabulate? Score this separately because it's often the starkest difference between the two systems — retrieval-free models are notoriously willing to guess.
Retrieval precision@k. For RAG only: of the chunks retrieved, how many were actually relevant to the question? Low precision with high-quality answers means the model is compensating for a weak retriever, which is a fragile setup that will degrade as the corpus grows.
Latency and cost per query. Not a quality metric, but it belongs in the same table. A RAG system that wins correctness by 4 points but costs 3x the latency needs that trade-off stated plainly, not hidden in an appendix.

Weight these per bucket rather than averaging everything into one score. A single blended "quality score" across all four buckets will wash out the exact signal you're trying to find, which is usually bucket-specific: RAG should dominate multi-hop and factual-lookup, retrieval-free should hold even on general-knowledge, and abstention accuracy tells you which system is safer to ship.

Using LLM-as-a-Judge to scale the comparison

Manually reading a few hundred answer pairs for four dimensions each is not realistic past the first pilot. This is where an LLM-as-a-Judge setup earns its place — a second LLM call that scores each answer against a rubric, run at scale across your full comparison set.

The judge prompt needs to be more structured than "which answer is better," because pairwise preference alone doesn't tell you *why*. A better pattern is to score each answer independently on the dimensions from the previous section, then diff the scores:

JUDGE_PROMPT = """
You are evaluating an AI assistant's answer for correctness and faithfulness.

Question: {question}
Reference answer (if available): {reference_answer}
Retrieved context (if this is a RAG answer, otherwise "N/A"): {context}
Answer to evaluate: {answer}

Score the answer on each dimension from 1-5:
1. correctness: does it match the reference answer's facts?
2. faithfulness: is every claim supported by the retrieved context?
   (score 5 if there is no context and the answer is clearly marked
   as general knowledge, not asserted as document-grounded)
3. abstention: if the context/knowledge doesn't contain the answer,
   did the model correctly say so instead of guessing? (score 5 if
   not applicable — an answer was actually available)

Return strict JSON:
{{"correctness": <1-5>, "faithfulness": <1-5>, "abstention": <1-5>,
  "reasoning": "<one sentence per dimension>"}}
"""

def judge_answer(result, judge_client) -> dict:
    prompt = JUDGE_PROMPT.format(
        question=result.question,
        reference_answer=result.reference_answer or "N/A",
        context="\n\n".join(result.retrieved_chunks) if result.retrieved_chunks else "N/A",
        answer=result.answer,
    )
    response = judge_client.generate(user_prompt=prompt, temperature=0.0)
    return json.loads(response.text)

Two things matter for this to be trustworthy rather than theater. First, run the judge at temperature=0.0 and, ideally, average over 2-3 samples per answer — judges are noisier than people expect, and a single low-temperature call can still swing a point or two on borderline cases. Second, calibrate the judge against a small human-labeled slice before trusting it on the full set: take 30-40 answer pairs, score them yourself, run the judge on the same pairs, and check agreement. If the judge and your human scores disagree by more than a point on average, the rubric needs tightening before you scale it — usually this means being more specific about what counts as "faithful" versus "correct," since judges conflate the two if the prompt doesn't force them apart.

Once calibrated, run the judge across the full comparison set for both systems and aggregate by bucket. The output you want is a table like this, per bucket, per system: mean correctness, mean faithfulness, mean abstention accuracy, mean latency. That table is the actual deliverable — it's what tells a stakeholder "RAG wins on factual lookups by 1.2 points, loses nothing on general knowledge, and costs 400ms more per query, so ship it for the support-docs use case but skip it for the FAQ chatbot."

Reading the results: what a win actually looks like

Once the scores are in, resist three common misreadings.

The first is treating a marginal win as a real win. If RAG beats retrieval-free by 0.3 points on a 5-point correctness scale, that's within judge noise for most rubrics — not a result to build a roadmap on. Look for gaps of at least a full point, or better, look at the raw disagreement rate: what fraction of questions did the two systems answer differently, and in which direction.

The second is ignoring bucket-level results in favor of the average. A RAG system that wins big on multi-hop questions but loses on general-knowledge questions (because retrieval sometimes surfaces an irrelevant or outdated chunk that displaces correct parametric knowledge) is a system worth shipping with a routing layer — send obviously general questions around retrieval, send corpus-specific questions through it. You'd never see that if you only looked at the blended score.

The third is forgetting the failure mode unique to RAG: confidently wrong because of bad retrieval. When RAG loses on the factual bucket, always check retrieved_chunks before concluding "RAG doesn't help here." Two very different bugs produce the same low score: the retriever pulled the wrong chunk (a chunking or embedding problem, fixable by adjusting chunk size, overlap, or embedding model), or the retriever pulled the right chunk but the model ignored it in favor of its own parametric guess (a prompting problem, fixable by strengthening the instruction to prioritize context). Your eval harness should make it trivial to tell these apart — this is exactly why the harness above stores retrieved chunks alongside every answer.

A worked example: support documentation Q&A

Say you're building a support bot for a SaaS product with a 200-page help center. You run the four-bucket comparison with 20 questions per bucket, 160 total, each answered by both systems, each scored by the judge across three dimensions.

A realistic pattern looks like this: on the factual-lookup bucket, RAG scores meaningfully higher on correctness because the model has no way to know your product's specific pricing tiers or feature gating without the source text. On the general-knowledge bucket ("how do I reset my password" phrased generically), both systems score similarly, because the model has seen enough generic SaaS documentation patterns during training to answer reasonably even without retrieval. On the multi-hop bucket, results are more mixed — RAG's advantage narrows or disappears if your chunking strategy splits related facts across chunks that don't both get retrieved, which is a signal to fix chunking before concluding "RAG doesn't help with synthesis questions." On the out-of-corpus bucket, the gap in abstention accuracy is often the most decisive finding: retrieval-free systems tend to guess plausibly wrong answers to questions about your product because the phrasing pattern-matches something the model saw in training for a different product, while a properly prompted RAG system that finds no relevant chunks says "I don't have information about that" far more reliably.

The conclusion from a comparison shaped like this usually isn't "RAG wins" or "RAG loses" — it's "RAG earns its cost on factual and abstention-critical questions, adds no value on generic ones, and is only as good as the chunking on synthesis questions." That's a nuanced, actionable finding, and it's only visible because the eval was run bucket-by-bucket with faithfulness and abstention scored separately from raw correctness.

Common mistakes that invalidate the comparison

A few pitfalls show up repeatedly when teams run this kind of eval for the first time.

Different prompts for each system beyond the necessary context difference. If your RAG system prompt is meaningfully better-engineered than your retrieval-free one (more explicit instructions, better formatting guidance), you're measuring prompt quality, not retrieval value. Keep everything except the context-injection step identical.
Testing only on questions retrieval was designed to answer. If your test set is built from documents you know are in the corpus, you'll never see the abstention failures that matter most in production, where users ask about things that aren't documented at all.
Ignoring retriever-only metrics. A RAG system can look bad in end-to-end scoring for a reason that has nothing to do with generation — the retriever just isn't finding the right chunks. Precision@k and recall@k on a held-out set of question-to-source mappings will tell you this before you waste time tuning prompts.
Single-run judging with no calibration. An uncalibrated judge is worse than a coin flip dressed up as data, because it looks authoritative. Always validate against a human-labeled sample first.
Averaging across buckets that don't belong in the same average. As covered above, blend scores only after you've looked at the bucket breakdown, never instead of it.

Wrapping up: the eval is the deliverable, not the RAG system

The instinct to reach for RAG whenever grounding is the goal is understandable, but it should be a hypothesis you test, not a default you ship. The side-by-side harness described here — four question buckets, a shared prompt scaffold, per-dimension scoring, bucket-level aggregation — takes a day or two to build and will save weeks of infrastructure investment on a retrieval pipeline that turns out not to move the needle for your specific corpus and question mix.

The core discipline is simple to state and easy to skip under deadline pressure: never trust a single blended quality number, always separate correctness from faithfulness from abstention, always keep the retrieved chunks so you can debug a loss instead of just observing it, and always calibrate your LLM-as-a-Judge against a human-labeled sample before you let it score thousands of answers unsupervised. Do that, and the question "should we add RAG" stops being a matter of instinct and becomes something you can actually answer with a number.