RAG

Self-RAG and Corrective RAG: Agents That Check Their Own Retrieval

Pramod Dutta · May 4, 2026 · 15 min read

Most RAG pipelines are built on an assumption that quietly falls apart in production: that whatever the retriever pulls back is good enough to answer with. Embed the query, run a similarity search, stuff the top-k chunks into the prompt, and let the model generate. It works beautifully in demos because demo questions are usually phrased close to the training documents. Then a real user asks something slightly off-topic, the retriever returns four confidently irrelevant chunks, and the model happily hallucinates an answer built on top of them. Nobody in the pipeline ever asked "was this retrieval actually any good?" That single missing question is what self-RAG and corrective RAG are built to answer, and it's the difference between a RAG system that looks impressive in a walkthrough and one that survives contact with real traffic.

Why "retrieve then generate" breaks down

The standard RAG loop has no feedback path. A query goes in, chunks come out, and the generator is never given permission to say "these chunks don't help me." Even worse, the generator is *trained* to be helpful, so it will construct a plausible-sounding answer out of whatever context it was handed, relevant or not. This creates three recurring failure modes that show up in almost every production RAG system I've debugged:

Silent irrelevance: the retriever returns chunks that are topically adjacent but don't actually contain the answer, and the model fills the gap with its own prior knowledge, uncredited.
Stale or wrong documents: the vector index has an outdated version of a policy or a doc that was since corrected, and there's no mechanism to notice the retrieved content contradicts more current information.
Overconfidence: the model states an answer with the same tone whether it retrieved a perfect chunk or complete noise, because nothing in the pipeline scores retrieval quality before generation.

Self-RAG and corrective RAG both attack this problem, but from different angles. Self-RAG bakes a self-evaluation habit directly into the generation process using special control tokens. Corrective RAG (often abbreviated CRAG) treats retrieval quality as an external, explicit gate that can trigger a repair action — a web search, a query rewrite, a fallback source — before generation even starts. In practice, the systems I've shipped use ideas from both, and understanding them as complementary rather than competing is the key insight for anyone building agents on top of retrieval.

What self-RAG actually adds to the pipeline

Self-RAG, as introduced in the original research from the University of Washington and Allen Institute for AI, trains a language model to emit special reflection tokens alongside its normal output. Instead of just generating text, the model interleaves decisions like:

Retrieve or not: given the current query and what's already been generated, does the model even need to pull more context, or can it answer from what it already has?
IsRel (is relevant): for each retrieved passage, is it actually relevant to the query?
IsSup (is supported): does the generated statement follow from the retrieved passage, or is the model going beyond what the evidence says?
IsUse (is useful): independent of relevance and support, is this response actually useful to the user?

The clever part is that these aren't bolted-on classifiers running in a separate pass — they're tokens the model itself was trained to produce, which means the judgment is part of the same forward pass as generation. When you don't have access to a model fine-tuned this way (which is most of us, since off-the-shelf frontier models don't ship with these control tokens baked in), you can approximate the same behavior with a lightweight critique step layered on top of a standard RAG call.

Here's a simplified version of that pattern, using a critique-and-regenerate loop instead of native reflection tokens:

def self_rag_answer(query, retriever, llm, max_retries=2):
    passages = retriever.search(query, k=5)

    for attempt in range(max_retries + 1):
        relevant = [p for p in passages if is_relevant(query, p, llm)]

        if not relevant:
            # nothing usable came back, try a reformulated query
            query = rewrite_query(query, llm)
            passages = retriever.search(query, k=5)
            continue

        draft = generate_answer(query, relevant, llm)

        if is_supported(draft, relevant, llm) and is_useful(draft, query, llm):
            return draft

        # answer wasn't grounded or useful, tighten the retrieval and retry
        passages = retriever.search(query, k=8)

    return generate_answer(query, relevant, llm, fallback=True)


def is_relevant(query, passage, llm):
    prompt = f"Query: {query}\nPassage: {passage.text}\nIs this passage relevant to answering the query? Answer yes or no."
    return llm.complete(prompt).strip().lower().startswith("yes")


def is_supported(answer, passages, llm):
    context = "\n".join(p.text for p in passages)
    prompt = f"Context:\n{context}\n\nAnswer:\n{answer}\n\nIs every claim in the answer directly supported by the context? Answer yes or no."
    return llm.complete(prompt).strip().lower().startswith("yes")

This is obviously more expensive than a single retrieve-and-generate call — you're paying for extra LLM calls to grade relevance and support. But for anything higher-stakes than a toy chatbot, that cost is usually worth it. I've used variations of this pattern in support-ticket assistants where a wrong answer costs more in customer trust than three extra API calls ever would.

One detail that trips people up the first time they implement this: the rewrite_query step needs its own prompt discipline, or it just paraphrases the original query and gets the same bad results back. A useful trick is to ask the model to generate the rewrite as if it were a domain expert who already knows the terminology the knowledge base uses, rather than the phrasing a customer would type. If a user asks "why did my payment fail," and the knowledge base talks about "transaction decline codes," a naive rewrite keeps saying "payment fail." A better rewrite prompt explicitly asks the model to guess the internal vocabulary the documents probably use, which closes a surprising amount of the semantic gap that embeddings alone don't bridge.

Corrective RAG: treating retrieval as a gate, not a given

Corrective RAG takes a different architectural stance. Rather than teaching the generator to second-guess itself token by token, CRAG inserts an explicit retrieval evaluator between the retriever and the generator. This evaluator scores the retrieved documents as a whole and routes the pipeline down one of three paths:

Correct: retrieval is good, refine the passages (strip noise, keep only the salient sentences) and generate normally.
Incorrect: retrieval is bad, discard it entirely and fall back to an external knowledge source, most commonly a live web search.
Ambiguous: retrieval is partially useful, blend the internal documents with supplementary web results before generating.

The elegance of CRAG is that it doesn't require retraining anything. You can build the evaluator as a small classifier, a fine-tuned lightweight model, or — in most practical implementations I've built — a well-prompted LLM call that just grades the retrieved set. The architecture looks like this:

def corrective_rag_answer(query, vector_retriever, web_search, llm):
    docs = vector_retriever.search(query, k=6)
    score = evaluate_retrieval(query, docs, llm)  # returns "correct" | "incorrect" | "ambiguous"

    if score == "correct":
        context = refine_documents(docs, query, llm)
    elif score == "incorrect":
        context = web_search.search(query)
    else:  # ambiguous
        refined = refine_documents(docs, query, llm)
        web_results = web_search.search(query)
        context = refined + web_results

    return generate_answer(query, context, llm)


def evaluate_retrieval(query, docs, llm):
    joined = "\n---\n".join(d.text for d in docs)
    prompt = f"""Query: {query}
Retrieved documents:
{joined}

Grade the retrieved documents as one word: correct, incorrect, or ambiguous.
- correct: the documents fully answer the query
- incorrect: the documents are irrelevant or contradict what's being asked
- ambiguous: the documents are partially relevant but incomplete"""
    return llm.complete(prompt).strip().lower()


def refine_documents(docs, query, llm):
    # decompose each document into sentence-level "knowledge strips",
    # score each strip for relevance, and keep only the useful ones
    strips = []
    for doc in docs:
        for sentence in split_into_sentences(doc.text):
            if is_relevant(query, sentence, llm):
                strips.append(sentence)
    return strips

That refine_documents step is the part teams skip most often, and it's the one that actually earns CRAG its name. Instead of throwing whole chunks at the generator, you decompose retrieved passages into fine-grained "knowledge strips" and filter at the sentence level. A five-hundred-word chunk might have one useful sentence and four paragraphs of irrelevant boilerplate — passing the whole chunk dilutes the signal and gives the model more surface area to hallucinate connections that aren't there.

The web-search fallback deserves a closer look too, because it's the piece people implement laziest. It's tempting to treat "incorrect" as a trigger to fire off a generic web search API and hope for the best, but an ungrounded web search has exactly the same failure mode as the vector retriever did — it can return irrelevant or low-quality pages just as easily. The fallback source needs its own relevance filtering, ideally reusing the same is_relevant check you already built for the internal documents. I've also seen teams get much better mileage by scoping the fallback: instead of an open web search, point it at a curated set of sources (your own public documentation, a vendor's changelog, a specific subreddit or forum known to be reliable for the domain) rather than the open web. It keeps the corrective step from trading one unreliable source for another.

Building the retrieval evaluator without overengineering it

Teams new to corrective RAG often assume they need a trained classifier to grade retrieval quality, and then stall out trying to build a labeled dataset. In practice, an LLM-as-judge call is a perfectly good starting point, and it's what I recommend building first before ever considering a fine-tuned evaluator. The key is being specific about the grading rubric instead of asking a vague "is this good?" question.

A rubric that has worked well for me in production:

Does the retrieved content mention the specific entities in the query (product names, dates, people, error codes)?
Does it directly address the *type* of question being asked (a how-to question needs procedural content, not a definition)?
Is there a contradiction between documents that suggests staleness?
Would a domain expert consider this passage sufficient to answer the question unassisted?

Only after you've validated the LLM-as-judge approach against a set of hand-labeled examples — and only if latency or cost becomes a real constraint — does it make sense to distill that judge into a smaller, cheaper classifier. Most teams never need to take that step; the judge call is a fraction of the cost of the generation call itself.

Combining self-RAG and corrective RAG in one pipeline

These two approaches aren't mutually exclusive, and the strongest production systems I've built borrow from both. Think of corrective RAG as the outer loop that decides *what evidence gets into the context window at all*, and self-RAG-style critique as the inner loop that decides *whether the generated answer actually respects that evidence*. A combined pipeline looks roughly like this:

def combined_rag_answer(query, vector_retriever, web_search, llm):
    # Corrective layer: gate and repair retrieval
    docs = vector_retriever.search(query, k=6)
    grade = evaluate_retrieval(query, docs, llm)

    if grade == "incorrect":
        context = web_search.search(query)
    elif grade == "ambiguous":
        context = refine_documents(docs, query, llm) + web_search.search(query)
    else:
        context = refine_documents(docs, query, llm)

    # Self-RAG layer: generate, then critique before returning
    draft = generate_answer(query, context, llm)

    if not is_supported(draft, context, llm):
        # the generator drifted beyond the evidence, force a stricter regeneration
        draft = generate_answer(
            query,
            context,
            llm,
            instruction="Only state facts explicitly present in the context. If unsure, say so."
        )

    return draft

The corrective layer prevents garbage from entering the context window in the first place. The self-critique layer catches the cases where the generator still wanders off despite having decent evidence — which happens more often than people expect, especially with longer context windows where the model has more room to blend retrieved facts with its own priors.

How chunking strategy changes what "checking retrieval" even means

A detail that's easy to overlook: self-RAG and corrective RAG behave very differently depending on how your documents were chunked in the first place. If your chunking strategy produces large, loosely-bounded chunks (say, splitting purely on a fixed token count with no regard for section boundaries), then even a "correct" retrieval grade is misleading, because the useful sentence is buried inside a chunk full of unrelated material. The refine_documents sentence-level filter partially compensates for this, but it's treating a symptom rather than the cause.

Smaller, semantically coherent chunks — split on headings, paragraphs, or logical sections rather than a blind token window — make the relevance grading step both cheaper and more accurate, because the judge model is evaluating something closer to a single idea instead of a grab-bag of loosely related sentences. In practice, I've found that pairing corrective RAG with chunk sizes in the 150-300 token range, with slight overlap to preserve context across boundaries, produces noticeably cleaner "correct" versus "incorrect" grades than trying to run the same evaluator over 1000-token chunks. The evaluator has less to reconcile, so its judgments are more decisive and its errors are easier to debug.

This also affects how you should think about the k in retriever.search(query, k=6). With larger chunks, teams often set k low because each chunk already has a lot of content. With smaller, cleaner chunks, it's usually better to retrieve more of them and let the relevance filter narrow things down, rather than relying on the embedding similarity ranking alone to make the cut. The self-checking layer is precisely what makes a higher initial k safe — without it, a wider retrieval just floods the generator's context with more noise.

Where this fits inside an agentic system

Both patterns become more valuable, not less, once you move from a single RAG call into a multi-step agent. An agent that plans several tool calls in sequence — search internal docs, check a database, call an API, then answer — compounds retrieval errors if none of the intermediate steps are checked. A bad retrieval at step one silently corrupts every downstream decision. This is exactly the kind of failure mode we spend real time on in Advanced AI Agents, because the moment you give an agent autonomy to decide *when* to retrieve and *what* to do with the result, you also need to give it the ability to notice when that decision went wrong.

Concretely, wiring self-RAG and corrective RAG into an agent means:

Adding a "confidence" or "grounded" flag to every tool result that flows through the agent's working memory, not just the final answer.
Letting the agent's planner treat an "incorrect" retrieval grade as a legitimate reason to try a different tool, not just retry the same search with the same query.
Logging *why* a retrieval was graded poorly (irrelevant, stale, contradictory) so you can debug retrieval quality across sessions instead of only debugging final answers.

Common implementation mistakes

A few patterns show up repeatedly when teams first implement these techniques, and they're worth naming explicitly:

Grading the whole answer instead of grading claims: a support-style critique of "is this answer good?" is too coarse. Break the generated answer into individual claims and check each one against the retrieved context separately — one unsupported sentence shouldn't sink an otherwise well-grounded paragraph, but it does need to be flagged or removed.
Using the same model temperature for critique and generation: generation can tolerate some creativity in phrasing, but critique steps should run at low or zero temperature. You want the relevance and support checks to be as deterministic as possible.
Retrying with the identical query: if retrieval failed, resubmitting the exact same query to the exact same index will fail the exact same way. Query rewriting — expanding acronyms, adding synonyms, or restructuring a question into a statement — is what actually improves the second attempt.
No ceiling on retries: without a max-attempts guard, a self-RAG loop can spin forever on a genuinely unanswerable question. Always cap retries and have an honest fallback response ("I don't have reliable information on this") rather than looping until the answer sounds confident.
Ignoring cost math early: every relevance check, support check, and retrieval grade is an additional LLM call. For high-volume, low-stakes queries (like a FAQ bot), this overhead might not be justified. Reserve the full self-RAG-plus-CRAG treatment for use cases where a wrong answer is expensive — medical, legal, financial, or anything customer-facing where trust is the product.

Evaluating whether it's actually working

Once the pipeline is live, you need a way to know if the self-checking is earning its cost. The metric that matters most isn't "how often does the model retry" — that just measures activity. What matters is:

Groundedness rate: of the answers that pass the support check, what fraction, when manually spot-checked, actually do trace back to the cited passages? This catches cases where your is_supported judge is too lenient.
Correction rate: how often does the corrective layer route to web search or ambiguous-blend, versus straight-through "correct"? A rate that's near zero suggests your evaluator isn't actually discriminating between good and bad retrieval — it might just be rubber-stamping everything as correct.
Downstream error reduction: track the rate of user-reported wrong answers before and after adding the self-checking layer. This is the only metric that ties the engineering effort back to something a stakeholder cares about.

Build a small evaluation set of queries where you already know the retriever will struggle — ambiguous phrasing, questions about deprecated features, questions the internal knowledge base simply doesn't cover — and run it through the pipeline regularly. This adversarial-by-design test set will tell you far more about whether corrective RAG is doing its job than any general-purpose benchmark, because it's specifically probing the failure modes these techniques exist to catch.

Self-RAG and corrective RAG aren't exotic research curiosities anymore — they're becoming the baseline expectation for any RAG system that has to be right more often than it has to be fast. The core habit both techniques instill is the same: don't let a generation step trust its input blindly. If you're still early in the RAG learning curve and these ideas about relevance grading and retrieval evaluation feel like a lot to take in at once, our Introduction to RAG course builds up from plain vector search to exactly this kind of self-correcting pipeline, one deliberate layer at a time.