Building a RAG Pipeline with LlamaIndex: A Complete Walkthrough

Why your first RAG demo works and your second one doesn't

Everyone's first retrieval-augmented generation demo looks the same: load a PDF, chunk it, embed it, ask a question, get a shockingly good answer. It takes about twenty minutes and it feels like magic. Then you point the same pipeline at your actual documentation set — a few hundred files, inconsistent formatting, tables buried inside paragraphs — and the answers get vague, the citations point to the wrong section, and half your queries return "I don't have enough information to answer that."

The gap between the demo and the real system isn't the LLM. It's everything around it: how you load documents, how you chunk them, what you embed, how you retrieve, and what you do with what you retrieve before it ever reaches the model. LlamaIndex was built specifically to give you control over that middle layer without forcing you to write a retrieval engine from scratch.

This walkthrough builds a complete RAG pipeline in LlamaIndex, piece by piece, explaining the decision at each stage instead of just handing you a finished script. By the end you'll have a working pipeline over a local document set, an understanding of where the failure modes live, and a sense of what to reach for when the naive version isn't good enough.

Setting up the environment

Start with a clean virtual environment. LlamaIndex ships as a set of packages — a core package plus integrations for vector stores, LLMs, and embedding providers — so you install only what you need.

python -m venv rag-env
source rag-env/bin/activate
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai
pip install llama-index-vector-stores-chroma chromadb

We're using OpenAI for the LLM and embeddings here because it keeps the example short, and Chroma as a local vector store so you can run everything without provisioning external infrastructure. LlamaIndex has first-class integrations for Anthropic's Claude models, Cohere embeddings, Pinecone, Weaviate, and a long list of others — swapping providers later is mostly a matter of changing which classes you import.

Set your API key as an environment variable rather than hardcoding it:

export OPENAI_API_KEY="sk-..."

With the environment ready, the next question is what you're actually indexing.

Loading your documents

LlamaIndex's SimpleDirectoryReader handles the common cases out of the box: PDFs, Word docs, Markdown, plain text, even CSVs. For anything unusual — Notion exports, a database table, a Slack archive — there's a reader in the LlamaHub ecosystem, or you write a small custom loader.

from llama_index.core import SimpleDirectoryReader

documents = SimpleDirectoryReader(
    input_dir="./docs",
    recursive=True,
    required_exts=[".md", ".pdf", ".txt"],
).load_data()

print(f"Loaded {len(documents)} documents")
print(documents[0].metadata)

Each item in documents is a Document object holding raw text plus metadata — file path, creation date, page number for PDFs. That metadata matters more than people expect. If you're building a support-docs assistant and a user asks "what does the June changelog say," you need the file name and date attached to each chunk, or the retriever has no way to filter on it later.

A detail that trips people up: SimpleDirectoryReader treats each file as one Document, not one chunk. Chunking happens in the next stage, when you build the index. If you load a 40-page PDF, you get one Document object containing the whole thing until you tell LlamaIndex how to split it.

You can also attach metadata manually, which is worth doing if your source system has structure the reader can't infer:

for doc in documents:
    doc.metadata["source_type"] = "internal_docs"
    doc.metadata["indexed_at"] = "2026-07-03"

Chunking strategy: the decision that matters most

If there's one stage in a RAG pipeline where more thought pays off disproportionately, it's chunking. Chunk too large and you dilute relevance — the embedding represents five ideas at once and matches none of them well. Chunk too small and you lose context — a paragraph that only makes sense next to the sentence before it now stands alone.

LlamaIndex calls this a NodeParser. The default is SentenceSplitter, which respects sentence boundaries while targeting a token count:

from llama_index.core.node_parser import SentenceSplitter

splitter = SentenceSplitter(
    chunk_size=512,
    chunk_overlap=50,
)

nodes = splitter.get_nodes_from_documents(documents)
print(f"Created {len(nodes)} nodes from {len(documents)} documents")

chunk_overlap matters because it prevents a sentence from being cleanly severed at a chunk boundary — the last 50 tokens of one chunk repeat as the first 50 of the next, so context isn't lost right at the seam.

For structured content, a flat sentence splitter isn't always the right tool. If your source is Markdown with clear headers, MarkdownNodeParser keeps sections intact along header boundaries, which tends to produce chunks that are semantically coherent rather than just token-count coherent:

from llama_index.core.node_parser import MarkdownNodeParser

md_parser = MarkdownNodeParser()
nodes = md_parser.get_nodes_from_documents(documents)

For code, there's CodeSplitter, which parses along function and class boundaries instead of blindly counting tokens. If a meaningful fraction of your corpus is source code, use it — a sentence splitter will happily cut a function in half.

A practical rule of thumb: start with chunk_size=512 and chunk_overlap=50 for prose, evaluate retrieval quality on a set of real questions, and only tune from there. Don't guess at chunk size in the abstract — measure it against actual queries.

It also helps to inspect a handful of nodes manually before you move on, rather than trusting the parser blindly:

for node in nodes[:3]:
    print("---")
    print(node.text[:300])
    print("metadata:", node.metadata)

You're looking for two failure signs. First, chunks that cut off mid-sentence or mid-table, which usually means the splitter isn't a good fit for that content type. Second, chunks that are mostly boilerplate — page headers, navigation text, repeated disclaimers — which waste embedding budget and crowd out real content in the retrieved set. If you see a lot of the second problem, it's worth adding a cleanup pass that strips boilerplate before chunking rather than trying to fix it downstream with a bigger top-k.

Building the index

Once you have nodes, you embed them and store the vectors. LlamaIndex's VectorStoreIndex handles both steps, calling out to whatever embedding model and vector store you've configured.

from llama_index.core import VectorStoreIndex, StorageContext, Settings
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb

Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("docs")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = VectorStoreIndex(
    nodes,
    storage_context=storage_context,
)

Settings is a global configuration object in LlamaIndex — set the embedding model and LLM once, and every downstream component picks them up without you passing them around explicitly. This is convenient, but it's also a common source of confusion in larger codebases where you forget you set something globally three files ago. In production, prefer being explicit and passing embed_model and llm directly where you can, especially if you run more than one pipeline in the same process.

Persisting to Chroma means you don't re-embed on every run — a real concern once your corpus is large enough that re-embedding costs real money and real time. If you're iterating on chunking strategy, embed once, save the vector store, and reload it:

index = VectorStoreIndex.from_vector_store(vector_store)

For genuinely small corpora — a few dozen documents used in a prototype — an in-memory index without a persistent vector store is fine and simpler:

index = VectorStoreIndex(nodes)
index.storage_context.persist(persist_dir="./storage")

Querying the index: from naive to reranked

The simplest possible query interface is index.as_query_engine(). It retrieves the top-k most similar nodes by cosine similarity, stuffs them into a prompt template, and calls the LLM.

from llama_index.llms.openai import OpenAI

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1)

query_engine = index.as_query_engine(similarity_top_k=5)
response = query_engine.query("How does the refund policy handle partial shipments?")

print(response)
for node in response.source_nodes:
    print(node.metadata.get("file_name"), node.score)

response.source_nodes is the piece people forget to inspect until something goes wrong. It tells you exactly which chunks the LLM was given and how similar each was scored — invaluable when a user reports a wrong answer and you need to know whether retrieval or generation failed.

Pure vector similarity has a known weakness: it's good at "these texts mean similar things" and bad at "this text contains this exact term." If a user asks about an exact SKU number or an error code, semantic similarity alone may not surface the chunk containing it, because embeddings capture meaning, not literal string overlap. This is where reranking earns its keep.

A common, effective pattern is a two-stage retrieval: pull a wider set of candidates cheaply with vector search, then rerank them with a more expensive, more accurate model before handing the top few to the LLM.

from llama_index.core.postprocessor import SentenceTransformerRerank

reranker = SentenceTransformerRerank(
    model="cross-encoder/ms-marco-MiniLM-L-6-v2",
    top_n=3,
)

query_engine = index.as_query_engine(
    similarity_top_k=15,
    node_postprocessors=[reranker],
)

response = query_engine.query("How does the refund policy handle partial shipments?")

Here the vector store returns 15 candidates, the cross-encoder reranker scores each candidate against the query directly (rather than comparing precomputed embeddings), and only the top 3 go into the final prompt. Cross-encoders are slower per-pair than embedding similarity but far more accurate, which is exactly why the two-stage pattern exists — you use the cheap method to narrow the field and the expensive method to pick the winners.

Adding metadata filtering for precision

Reranking improves relevance within the semantic search space. Metadata filtering constrains the search space itself, which is often the bigger lever when your corpus spans multiple sources, time periods, or access levels.

from llama_index.core.vector_stores import MetadataFilters, MetadataFilter, FilterOperator

filters = MetadataFilters(
    filters=[
        MetadataFilter(
            key="source_type",
            value="internal_docs",
            operator=FilterOperator.EQ,
        ),
    ]
)

query_engine = index.as_query_engine(
    similarity_top_k=5,
    filters=filters,
)

This is essential once you have multiple document classes in one index — say, public help-center articles and internal engineering runbooks. Without a filter, a semantically similar internal doc can outrank a more appropriate public one, and vice versa, depending on wording. Tagging documents with metadata at load time, as shown earlier, is what makes this filtering possible later. It's much easier to add metadata upfront than to retrofit it after you've already built and persisted an index.

Structuring the response with a custom prompt template

The default response synthesis prompt in LlamaIndex is generic — it works, but it doesn't know your domain's conventions. For most production use cases, you want to control the exact instructions the LLM receives about how to use the retrieved context.

from llama_index.core import PromptTemplate

qa_prompt_template = PromptTemplate(
    "Context information is below.\n"
    "---------------------\n"
    "{context_str}\n"
    "---------------------\n"
    "Given the context above and not prior knowledge, answer the question. "
    "If the context does not contain the answer, say you don't have "
    "enough information rather than guessing. Cite the source file name "
    "for any claim you make.\n"
    "Question: {query_str}\n"
    "Answer: "
)

query_engine = index.as_query_engine(
    similarity_top_k=5,
    node_postprocessors=[reranker],
)
query_engine.update_prompts(
    {"response_synthesizer:text_qa_template": qa_prompt_template}
)

response = query_engine.query("What is the maximum refund window for partial shipments?")
print(response)

The instruction "say you don't have enough information rather than guessing" is doing real work here. Without it, LLMs will confidently synthesize a plausible-sounding answer from context that only partially covers the question, and it reads exactly like a correct answer until someone checks it against the source. Explicitly permitting the model to decline is one of the cheapest reliability improvements you can make to a RAG system, and it costs nothing but a sentence in the prompt.

Evaluating retrieval quality, not just eyeballing answers

It's tempting to judge a RAG pipeline by asking it a few questions and seeing if the answers look right. That approach doesn't scale and it doesn't catch regressions when you change chunking or swap embedding models. LlamaIndex has a built-in evaluation module for exactly this.

from llama_index.core.evaluation import RelevancyEvaluator, FaithfulnessEvaluator

relevancy_evaluator = RelevancyEvaluator(llm=Settings.llm)
faithfulness_evaluator = FaithfulnessEvaluator(llm=Settings.llm)

query = "How does the refund policy handle partial shipments?"
response = query_engine.query(query)

relevancy_result = relevancy_evaluator.evaluate_response(query=query, response=response)
faithfulness_result = faithfulness_evaluator.evaluate_response(response=response)

print("Relevant:", relevancy_result.passing)
print("Faithful (grounded in source):", faithfulness_result.passing)

FaithfulnessEvaluator checks whether the generated answer is actually supported by the retrieved context — it catches hallucination even when the answer sounds fluent and confident. RelevancyEvaluator checks whether the retrieved context and the answer are actually relevant to the question asked, which catches cases where retrieval pulled the wrong chunks entirely.

Build a small evaluation set — twenty to fifty representative questions with known-good answers or known-good source documents — and run it every time you change a meaningful parameter: chunk size, embedding model, top-k, reranker. Treat it the same way you'd treat a test suite for application code. Without it, you're tuning a pipeline by vibes, and vibes don't catch the regression where your new chunking strategy silently drops the one paragraph that mattered.

Handling larger corpora: from single index to routed retrieval

Everything above assumes one corpus and one retrieval strategy. Once you're indexing multiple distinct document collections — product docs, legal contracts, support tickets — a single flat index starts to strain, because a query about a contract clause and a query about a UI bug shouldn't be searched the same way.

LlamaIndex's RouterQueryEngine lets you maintain separate indexes per collection and route each incoming query to the right one, using the LLM itself to decide which index is relevant:

from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.tools import QueryEngineTool
from llama_index.core.selectors import LLMSingleSelector

docs_tool = QueryEngineTool.from_defaults(
    query_engine=docs_index.as_query_engine(),
    description="Answers questions about product documentation and how-to guides.",
)

legal_tool = QueryEngineTool.from_defaults(
    query_engine=legal_index.as_query_engine(),
    description="Answers questions about contracts, terms, and legal policies.",
)

router_engine = RouterQueryEngine(
    selector=LLMSingleSelector.from_defaults(),
    query_engine_tools=[docs_tool, legal_tool],
)

response = router_engine.query("What's the cancellation clause in the enterprise contract?")

This is a meaningfully different architecture from a single vector index, and it's worth reaching for only once you actually have distinct document types with distinct retrieval needs — introducing it prematurely just adds a routing decision that can itself go wrong. Start with one index, watch where it breaks, and split only when the evidence tells you to.

Common pitfalls worth watching for

A few failure modes show up repeatedly across real deployments, and they're worth naming explicitly:

• Re-embedding on every run. If you rebuild the index from scratch on each deploy instead of persisting and updating incrementally, you burn embedding API calls and time for no benefit. Persist the vector store and use insert() for new documents.
• Ignoring `source_nodes`. When an answer is wrong, the fastest diagnosis is checking what was actually retrieved. If the right chunk wasn't in source_nodes, the bug is in retrieval, not generation — don't waste time tweaking the prompt.
• One chunk size for all content types. A 512-token chunk that works for prose will mangle a table or a code block. Route different content types through different parsers.
• No evaluation set. Changing chunk size or top-k without measuring the effect on a fixed set of test questions means you can't tell improvement from regression.
• Skipping metadata at load time. Filtering and citation both depend on metadata that's much cheaper to attach when you first load documents than to backfill later.

Where to go from here

This pipeline — load, chunk, embed, retrieve, rerank, filter, evaluate — covers the core mechanics that every production RAG system relies on, whether it's built with LlamaIndex, LangChain, or a custom retrieval layer. The specific API calls will vary, but the sequence of decisions (how to split documents, how to score relevance, how to keep the model honest about what it doesn't know) is the same regardless of framework.

If the concepts here — embeddings, vector similarity, why retrieval-augmented generation exists in the first place — felt like they needed more grounding, that's exactly what our Introduction to RAG course on teachyou.ai covers from first principles before you touch a framework at all. It's the natural starting point if you want the theory solid before you build the next version of this pipeline on your own documents.