8 Types of RAG Architectures Explained: Naive to Agentic RAG

Every team that ships a RAG system starts in the same place: chunk some documents, embed them, stuff the top-k matches into a prompt, and call it retrieval-augmented generation. It works, right up until it doesn't — the model cites the wrong policy version, misses a fact that's split across two chunks, or confidently answers from a document that was never actually relevant. At that point most teams discover that "RAG" isn't one architecture. It's a spectrum of techniques, each solving a specific failure mode of the one before it, at the cost of more latency, more infrastructure, or more engineering complexity.

This article walks through 8 types of RAG architectures you'll actually encounter in production systems, from the naive baseline to fully agentic pipelines, plus the long-context alternative that sometimes makes RAG unnecessary. For each one: what it is, when it earns its keep, and what it costs you. If you want the hands-on version of this — building each of these patterns end to end — that's exactly what we cover in Introduction to RAG.

1. Naive RAG

Naive RAG is the reference implementation everyone builds first: chunk documents into fixed-size windows, embed each chunk with a single embedding model, store the vectors in a vector database, and at query time embed the user's question, run a cosine-similarity (or dot-product) nearest-neighbor search, and paste the top-k chunks into the LLM's context window alongside the question.

The pipeline looks like this:

Ingest:  documents → chunk → embed → store in vector DB
Query:   question → embed → vector search (top-k) → stuff into prompt → LLM answer

There's nothing wrong with this as a starting point. It's fast to build, cheap to run, and for narrow, well-scoped knowledge bases — a single product's FAQ, a small internal wiki — it's often good enough. The failure modes show up as your corpus grows and your queries get harder: semantic search alone doesn't understand exact keyword matches (part numbers, error codes, acronyms), a single embedding per chunk can't represent every way a chunk might be relevant, and there's no correction step if the retrieved chunks are mediocre — the LLM just does its best with whatever it's handed.

When to use it: prototypes, small and homogeneous knowledge bases, internal tools where "good enough" retrieval is fine, or as the control group you benchmark every fancier architecture against.

Tradeoff: dead simple to build and cheap to operate, but retrieval quality is capped by embedding-similarity alone — it silently degrades as the corpus grows, and there's no mechanism to catch or correct bad retrievals before they reach the model.

2. Hybrid Search RAG (Dense + Sparse)

The first thing most teams add to naive RAG is hybrid search — combining dense vector retrieval with sparse keyword retrieval (typically BM25, the workhorse ranking function from classic information retrieval). Dense embeddings are excellent at capturing semantic meaning ("how do I cancel my subscription" matching a chunk titled "terminating your plan") but they're surprisingly bad at exact-match terms: SKUs, error codes, product names, legal clause numbers, anything where the literal string matters more than the concept. BM25 is the opposite — it's a term-frequency/inverse-document-frequency scoring method that nails exact and near-exact matches but has no notion of synonymy or paraphrase.

Hybrid search runs both retrievers in parallel and fuses the results, usually with Reciprocal Rank Fusion (RRF), which combines ranked lists without needing the raw scores to be on the same scale (cosine similarity and BM25 scores are not comparable numbers, so naive score-averaging is a common bug here).

from rank_bm25 import BM25Okapi

def hybrid_retrieve(query, dense_index, bm25_corpus, tokenized_docs, k=10, rrf_k=60):
    # Dense retrieval
    query_embedding = embed(query)
    dense_hits = dense_index.search(query_embedding, top_k=k)  # [(doc_id, rank), ...]

    # Sparse retrieval
    bm25 = BM25Okapi(tokenized_docs)
    tokenized_query = query.lower().split()
    bm25_scores = bm25.get_scores(tokenized_query)
    sparse_hits = sorted(enumerate(bm25_scores), key=lambda x: -x[1])[:k]

    # Reciprocal Rank Fusion
    fused_scores = {}
    for rank, (doc_id, _) in enumerate(dense_hits):
        fused_scores[doc_id] = fused_scores.get(doc_id, 0) + 1 / (rrf_k + rank + 1)
    for rank, (doc_id, _) in enumerate(sparse_hits):
        fused_scores[doc_id] = fused_scores.get(doc_id, 0) + 1 / (rrf_k + rank + 1)

    return sorted(fused_scores.items(), key=lambda x: -x[1])[:k]

Most managed vector databases (Weaviate, Qdrant, Elasticsearch/OpenSearch with vector plugins) now support hybrid search natively, so you often don't need to hand-roll RRF — but understanding what's happening under the hood matters when you're debugging why a query with a product code in it isn't surfacing the right chunk.

When to use it: any domain with technical vocabulary, product codes, proper nouns, or acronyms — which is most enterprise and technical documentation. This is close to a default-on upgrade over naive RAG for production systems.

Tradeoff: meaningfully better recall on keyword-heavy queries with modest added complexity (two indexes to maintain, a fusion step to tune), but it doesn't fix ranking quality within the fused list — it just gives you a better candidate set.

3. Reranking RAG

Hybrid search gives you a better *candidate set*, but the ordering of that candidate set is still driven by fast, approximate methods — vector similarity and term frequency. Reranking RAG adds a second-stage model that re-scores a smaller candidate list (say, the top 20-50 results from your first-stage retriever) using a model that can actually read the query and the passage together and judge relevance directly.

The pattern is a two-stage funnel: a cheap, fast, high-recall retriever narrows millions of documents down to a few dozen candidates, then an expensive, slow, high-precision reranker — typically a cross-encoder model — reorders those candidates before the top few go into the prompt. Cross-encoders process the query and each candidate passage together in a single forward pass, which lets them model interactions between query terms and passage terms that a bi-encoder (the kind of model that produces the embeddings used in step 1) fundamentally cannot capture, since bi-encoders score similarity between two vectors computed independently.

def retrieve_and_rerank(query, retriever, reranker, first_stage_k=50, final_k=5):
    # Stage 1: cheap, high-recall retrieval (dense, sparse, or hybrid)
    candidates = retriever.search(query, top_k=first_stage_k)

    # Stage 2: expensive, high-precision reranking (cross-encoder)
    pairs = [(query, doc.text) for doc in candidates]
    scores = reranker.predict(pairs)  # e.g. a cross-encoder model

    reranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return [doc for doc, score in reranked[:final_k]]

The reason you don't just run the cross-encoder over your whole corpus is cost: cross-encoders scale as O(query × candidate) pairs, which is fine for 50 candidates and prohibitive for a million documents. The two-stage design is what makes this tractable — cast a wide, cheap net, then spend your compute budget only on the passages that made the cut.

When to use it: whenever precision at the top of the list matters — which is nearly always, since LLMs are sensitive to irrelevant context polluting the prompt ("lost in the middle" effects, distraction from noisy chunks). It's a strong default addition once hybrid search is in place.

Tradeoff: noticeably improves precision and reduces irrelevant-context noise in the prompt, but adds a real latency cost (an extra model call per query) and another component to host, version, and monitor.

4. Query Rewriting and HyDE RAG

So far every architecture has assumed the user's query, as typed, is a good search query. It often isn't. Real user questions are conversational, underspecified, or reference prior turns ("what about the enterprise tier?" makes no sense as a standalone retrieval query). Query rewriting RAG inserts an LLM step before retrieval that transforms the raw query into one or more better search queries — resolving pronouns and references from conversation history, decomposing a compound question into sub-questions, or generating multiple paraphrases to retrieve against and merge (multi-query retrieval).

HyDE (Hypothetical Document Embeddings) is a specific, clever variant of this idea. Instead of embedding the user's question and searching for similar chunks, you first ask an LLM to *hypothesize an answer* to the question — even if that answer might be wrong or incomplete — and then embed that hypothetical answer and search with it. The intuition: a question and its answer are often written in different registers ("how do I fix a memory leak in Node?" vs. a passage that says "to resolve leaks, use --inspect and check for retained closures...") — questions and answer-shaped text don't always land close together in embedding space, but a hypothetical answer and the real answer usually do, because they're the same kind of text.

1. User query:        "How do I fix a memory leak in a Node service?"
2. LLM generates:     a plausible-sounding hypothetical answer/passage
3. Embed the hypothetical answer (not the original question)
4. Vector search using that embedding
5. Retrieve real chunks, discard the hypothetical text
6. Pass retrieved chunks + original query to the LLM for the final answer

Both techniques share a cost profile: they add an LLM call (sometimes several) purely for the purpose of improving what you search with, before the "real" generation call even happens.

When to use it: conversational/multi-turn interfaces where queries depend on context, question types that are structurally different from your source documents (short questions vs. long technical passages), or complex questions that benefit from decomposition into sub-queries.

Tradeoff: significantly better retrieval on ambiguous or conversational queries, at the cost of extra latency and LLM spend for the rewriting/hypothesis step — and HyDE specifically can misfire on niche or highly technical topics where the model's hypothesized answer is confidently wrong and steers retrieval away from the right chunks.

5. Graph RAG

Vector search — dense, sparse, or hybrid — treats your knowledge base as an unordered bag of chunks. It's fundamentally bad at questions that require connecting facts across multiple documents: "which vendors used by our EU subsidiaries also appear in the security incident reports from last year?" isn't a similarity-search question, it's a traversal question. Graph RAG addresses this by building a knowledge graph — entities as nodes, relationships as edges — typically extracted from your source documents using an LLM, and then answering queries by traversing that graph (directly, or by combining graph traversal with vector search over node/edge descriptions).

Microsoft's GraphRAG implementation popularized a specific flavor of this: extract entities and relationships from documents, cluster related entities into communities (using graph community-detection algorithms), have an LLM generate summaries of each community, and at query time either do local search (traverse from entities mentioned in the query) or global search (reason over community summaries) depending on whether the question is specific or broad-and-thematic.

The architecture is meaningfully heavier than anything above: you need an entity/relationship extraction pipeline (usually LLM-driven, which means it's slow and non-deterministic), a graph database or graph-capable store, and a query planner that decides whether and how to traverse. But it unlocks a genuinely different class of question — multi-hop reasoning across documents, and "summarize the themes across this whole corpus" queries that no chunk-level retriever can answer, because no single chunk contains the answer.

When to use it: domains where relationships between entities matter more than any single document's content — organizational knowledge, research literature, investigative/compliance workloads, or corpora where users ask holistic "what are the themes" questions rather than pointed lookups.

Tradeoff: unlocks multi-hop and thematic queries that vector-only RAG structurally cannot answer, at the cost of a much heavier ingestion pipeline (LLM-driven extraction is slow, expensive, and can hallucinate relationships) and real infrastructure investment in a graph store.

6. Agentic / Iterative RAG

Every architecture so far is a single pass: retrieve once, generate once. Agentic RAG turns retrieval into a loop controlled by the model itself. Instead of a fixed pipeline, the LLM is given retrieval as a tool it can call, decide whether the results are sufficient, reformulate its query and call it again, pull from a different source entirely, or decide it has enough to answer — all as part of an agent loop rather than a scripted sequence.

A minimal version of this is self-RAG or corrective RAG (CRAG) style patterns: after retrieval, an evaluation step (often the LLM itself, or a lightweight classifier) grades whether the retrieved chunks actually address the query. If they don't, the system retries with a rewritten query, expands the search to a different index or a web search tool, or explicitly tells the user it couldn't find a good answer instead of forcing a response from irrelevant context.

Loop (bounded by max_iterations):
  1. Agent decides: do I need to retrieve, and what should I search for?
  2. Call retrieval tool → get chunks
  3. Agent grades: are these chunks sufficient to answer the query?
     - Yes → generate final answer, exit loop
     - No  → reformulate query or switch retrieval source, go to step 2
  4. If max_iterations reached → answer with best available context,
     or explicitly say the answer isn't in the knowledge base

This is the pattern behind most "research agent" and "deep search" style products — the agent might issue five or six retrieval calls against different tools (a vector store, a SQL database, a web search API) before it's satisfied it has enough to answer, and it can recover from a bad first retrieval instead of being stuck with it.

When to use it: complex, open-ended questions where a single retrieval pass is unreliable, multi-source knowledge bases (internal docs + web + structured data), or any product where "I don't know" is a much better outcome than a confidently wrong answer built on irrelevant context.

Tradeoff: dramatically more robust to bad initial retrievals and capable of handling genuinely hard multi-step questions, but latency and cost scale with the number of loop iterations, and you need real engineering discipline around max-iteration caps and loop-termination logic or you'll ship a system that occasionally spins and burns tokens.

7. Multi-Vector / Parent-Document RAG

There's a tension baked into every chunking strategy: small chunks embed well (a tightly-scoped chunk produces a precise, unambiguous embedding) but generate poorly (a 200-token chunk often lacks the surrounding context the LLM needs to answer well). Large chunks generate better but embed worse (a long chunk covering multiple sub-topics produces a muddy, averaged-out embedding that matches everything and nothing precisely).

Multi-vector RAG — sometimes implemented as parent-document retrieval — resolves this by decoupling what you *embed and search* from what you *retrieve and hand to the LLM*. A common pattern: split documents into large "parent" chunks, then further split each parent into small "child" chunks. Embed and index only the children (since they produce cleaner, more targeted embeddings), but store a mapping from each child back to its parent. At query time, retrieve based on child-chunk similarity, but pass the full parent chunk — or the original full document — into the LLM's context.

A related variant embeds multiple *representations* of the same chunk instead of multiple chunk sizes: an LLM-generated summary of a chunk, a set of hypothetical questions the chunk could answer, and the raw chunk text itself, all embedded and indexed separately but all pointing back to the same source content. Whichever representation matches the query best, the underlying full chunk gets retrieved.

Ingest:
  document → parent chunks (large) → child chunks (small)
  embed child chunks only → store with parent_id reference
  store parent chunks separately (keyed by parent_id)

Query:
  question → embed → search child chunk index → get matching child_ids
  look up parent_id for each match → fetch full parent chunk
  pass parent chunks (not child chunks) to the LLM

When to use it: long-form or highly structured source documents (contracts, technical manuals, research papers) where a small chunk that matches well often lacks the surrounding context needed to answer correctly — you need the precision of small-chunk search with the context of large-chunk generation.

Tradeoff: gives you both precise retrieval and context-rich generation, but roughly doubles your storage and indexing complexity (two chunk granularities or multiple representations to manage, plus the parent-child mapping layer), and parent chunks can get large enough to eat significant context-window budget per retrieved result.

8. Long-Context "RAG-less" Retrieval

As context windows have grown into the hundreds of thousands and millions of tokens, a legitimate question emerged: for some use cases, why retrieve at all? If your entire knowledge base — or the relevant subset of it — fits in the model's context window, you can skip retrieval entirely and just stuff the whole corpus into the prompt on every call, letting the model's own attention mechanism do the "retrieval" implicitly.

This isn't really RAG at all — it's the baseline everyone should benchmark RAG systems against, because it's often a strong one for small-to-medium corpora. It sidesteps the entire retrieval-quality problem: no chunking strategy to tune, no embedding model to pick, no vector database to run, no recall/precision tradeoffs. The cost moves entirely to inference: every query reprocesses the full document set (though prompt caching on the provider side substantially reduces the repeated cost when the same context is reused across many queries), and there are real limits — a codebase or document set that's a few million tokens still won't fit, and even within the context window, models don't attend uniformly across very long inputs, so relevant information can still get under-weighted purely due to position.

Naive/Hybrid/Reranking/Agentic RAG:  question → retrieve relevant subset → generate
Long-context "RAG-less":              question → entire corpus in context → generate

When to use it: small-to-medium, relatively static knowledge bases where the whole thing plausibly fits in context (a single product's documentation, a codebase under a few hundred thousand tokens, a set of contracts for one deal); situations where retrieval errors are unacceptable and you'd rather pay the token cost than risk missing a relevant passage; or simply as your evaluation baseline before you justify the engineering cost of any retrieval pipeline.

Tradeoff: eliminates retrieval-quality risk entirely and radically simplifies the architecture, but cost and latency scale with corpus size on every call (even with caching), it hits a hard wall once your knowledge base exceeds the context window, and long-context attention still isn't perfectly uniform — position in the prompt can matter more than it should.

Which one should you actually build

If you're starting from zero, the realistic path is: build naive RAG first so you have a working baseline and an eval set, add hybrid search and reranking almost immediately since they're close to strictly-better upgrades for most domains, then reach for query rewriting/HyDE, graph RAG, multi-vector retrieval, or agentic loops only once you've measured a specific failure mode that those techniques actually fix. None of these architectures are mutually exclusive — most serious production RAG systems are a combination: hybrid retrieval feeding a reranker, wrapped in an agentic loop that can retry or fall back to a different source, over a parent-document index. And for a meaningful slice of use cases, the long-context baseline quietly wins and you never need most of this at all.

The pattern across every upgrade here is the same: identify exactly which failure mode you're solving for — missed keyword matches, noisy top-k results, ambiguous queries, cross-document reasoning, bad first retrievals, or a context/precision tradeoff in chunking — before you add the architecture that fixes it. Bolting on complexity without a measured reason is how RAG systems become expensive to run and hard to debug. If you want to build and evaluate each of these patterns yourself, with real corpora and real failure cases instead of toy examples, that's the core of what we teach in Introduction to RAG.