RAGLlamaIndexLangChainretrieval augmented generationpython frameworks

LlamaIndex vs LangChain for RAG

Pramod Dutta · Jul 2, 2026 · 11 min read

If you are choosing between LlamaIndex vs LangChain for RAG, the short answer is this: LlamaIndex is the better default when your core problem is indexing, chunking, and retrieving data well, while LangChain is the better default when RAG is one piece of a larger agentic workflow with tools, memory, and multi-step chains. Both can build a working RAG pipeline in under fifty lines of code. The difference shows up six weeks later, when you need hybrid search, custom node parsing, or a multi-agent system that happens to use retrieval as one of its tools.

This article walks through both frameworks side by side, with working code for the same RAG use case (a support-docs chatbot), then compares them on indexing flexibility, retrieval strategies, agent integration, evaluation, and production concerns like caching and observability.

What Each Framework Actually Optimizes For

LlamaIndex started as "GPT Index," a project purpose-built for one job: get unstructured data into a form an LLM can query accurately. Its abstractions (Document, Node, Index, QueryEngine) all exist to answer "how do I retrieve the right context." Everything about the API nudges you toward better chunking, better metadata, and better retrieval scoring.

LangChain started as a general orchestration layer for chaining LLM calls together. Its abstractions (Runnable, Chain, AgentExecutor, Tool) exist to answer "how do I compose steps, including but not limited to retrieval." RAG in LangChain is implemented as a retriever object plugged into a chain, alongside memory, tools, and output parsers.

Neither framework is "wrong" for RAG. They just made different bets about what the hard part is. If you think the hard part is retrieval quality, reach for LlamaIndex. If you think the hard part is gluing retrieval to five other capabilities (web search, calculators, SQL, multi-turn memory, human-in-the-loop approval), reach for LangChain.

Setting Up: The Same RAG Task in Both

Assume a folder of markdown support docs (./docs) and the goal of answering user questions with citations. Install both stacks in separate virtual environments so dependency trees do not collide.

python -m venv .venv-llamaindex
source .venv-llamaindex/bin/activate
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai

python -m venv .venv-langchain
source .venv-langchain/bin/activate
pip install langchain langchain-openai langchain-community chromadb

LlamaIndex version

import os
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

os.environ["OPENAI_API_KEY"] = "sk-..."

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine(similarity_top_k=4)
response = query_engine.query("How do I reset a user's password?")

print(response)
for node in response.source_nodes:
    print(node.metadata.get("file_name"), node.score)

LangChain version

import os
from langchain_community.document_loaders import DirectoryLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA

os.environ["OPENAI_API_KEY"] = "sk-..."

loader = DirectoryLoader("./docs", glob="**/*.md")
raw_docs = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
chunks = splitter.split_documents(raw_docs)

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(chunks, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True,
)

result = qa_chain.invoke({"query": "How do I reset a user's password?"})
print(result["result"])
for doc in result["source_documents"]:
    print(doc.metadata.get("source"))

Notice the difference already. LlamaIndex collapses loading, chunking, embedding, indexing, and querying into a handful of high-level calls with sane defaults. LangChain makes every step an explicit, swappable component: loader, splitter, embeddings, vector store, retriever, chain. That verbosity is a cost for a simple RAG bot, and an asset the moment you need to swap one piece (say, a different text splitter for code blocks) without touching the rest.

Indexing and Chunking Strategies

This is where LlamaIndex pulls ahead for pure RAG work. It ships purpose-built node parsers beyond generic character splitting:

from llama_index.core.node_parser import (
    SentenceSplitter,
    SemanticSplitterNodeParser,
    HierarchicalNodeParser,
)

# Splits on sentence boundaries with overlap, respects markdown structure
sentence_parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)

# Splits based on embedding similarity between sentences, not fixed size
semantic_parser = SemanticSplitterNodeParser(
    buffer_size=1,
    breakpoint_percentile_threshold=95,
    embed_model=Settings.embed_model,
)

# Builds parent/child chunk relationships for small-to-big retrieval
hierarchical_parser = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[2048, 512, 128]
)

The hierarchical parser is particularly useful: you retrieve small, precise chunks for matching, then automatically pull in the larger parent chunk for context when generating the answer. Doing this by hand in LangChain is possible with ParentDocumentRetriever, but it is one specific recipe rather than a first-class indexing primitive with multiple parser strategies to choose from.

LangChain's text splitters are good and cover the common cases (RecursiveCharacterTextSplitter, MarkdownHeaderTextSplitter, PythonCodeTextSplitter), but they are splitters, not indexers. LangChain does not have a strong opinion on how chunks relate to each other after splitting; that is left to whichever vector store or retriever you pick.

Retrieval Strategies: Hybrid, Auto-merging, Re-ranking

Both frameworks support hybrid search (dense + sparse/BM25) and re-ranking, but the ergonomics differ.

LlamaIndex bakes retrieval strategy into the query engine construction:

from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.postprocessor import SentenceTransformerRerank

retriever = VectorIndexRetriever(index=index, similarity_top_k=10)
reranker = SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_n=4)

query_engine = RetrieverQueryEngine.from_args(
    retriever=retriever,
    node_postprocessors=[reranker],
)

response = query_engine.query("What's the refund policy for annual plans?")

The node_postprocessors pattern is a clean, composable pipeline stage: filter by similarity threshold, rerank, deduplicate, all as stackable steps before the LLM ever sees the context.

LangChain's equivalent uses ContextualCompressionRetriever wrapping a base retriever with a compressor:

from langchain.retrievers import ContextualCompressionRetriever
from langchain_community.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

cross_encoder = HuggingFaceCrossEncoder(model_name="cross-encoder/ms-marco-MiniLM-L-6-v2")
compressor = CrossEncoderReranker(model=cross_encoder, top_n=4)

compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=vectorstore.as_retriever(search_kwargs={"k": 10}),
)

Functionally equivalent, similar line count. Where LangChain wins is retriever variety: EnsembleRetriever for combining BM25 and dense search with weighted fusion is a one-liner, and it interoperates cleanly with any Runnable-based chain.

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

bm25_retriever = BM25Retriever.from_documents(chunks)
bm25_retriever.k = 10

ensemble_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, vectorstore.as_retriever(search_kwargs={"k": 10})],
    weights=[0.4, 0.6],
)

LlamaIndex has hybrid search too, usually delegated to whichever vector store you connect (Qdrant, Weaviate, and Pinecone all support native hybrid mode through LlamaIndex's vector store integrations), which is arguably cleaner since the fusion happens at the database layer instead of in application code.

When RAG Is Part of a Bigger Agent

This is the clearest split point. If your support bot needs to retrieve docs, then decide whether to also check an order-status API, then decide whether to escalate to a human, LangChain's agent and graph tooling (LangGraph, built on top of LangChain) is purpose-built for that branching control flow.

from langchain.agents import create_tool_calling_agent, AgentExecutor
from langchain.tools.retriever import create_retriever_tool
from langchain_core.prompts import ChatPromptTemplate

retriever_tool = create_retriever_tool(
    retriever,
    "search_support_docs",
    "Search internal support documentation for policy and how-to answers.",
)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a support agent. Use tools to find accurate answers."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}"),
])

agent = create_tool_calling_agent(llm, [retriever_tool], prompt)
agent_executor = AgentExecutor(agent=agent, tools=[retriever_tool], verbose=True)

agent_executor.invoke({"input": "Can I get a refund on an annual plan I bought 40 days ago?"})

Add a second tool (an order-lookup API call) and a third (a human-escalation function), and the agent decides which to call, in what order, based on the query. This composability across many tool types, not just retrieval, is LangChain's strongest use case.

LlamaIndex has agents too (ReActAgent, FunctionAgent) and they work well when retrieval is the dominant tool and one or two others are bolted on. But once you are building a genuinely multi-tool, multi-step agent with conditional branching and persistent state across turns, LangGraph's explicit graph model (nodes, edges, state) is more legible than chaining LlamaIndex agent steps by hand.

Evaluation

Both frameworks ship evaluation helpers, and neither is a substitute for a dedicated eval tool like Ragas or DeepEval, but the built-ins are useful for quick sanity checks.

LlamaIndex:

from llama_index.core.evaluation import FaithfulnessEvaluator, RelevancyEvaluator

faithfulness_evaluator = FaithfulnessEvaluator(llm=Settings.llm)
relevancy_evaluator = RelevancyEvaluator(llm=Settings.llm)

response = query_engine.query("What is the refund window for monthly plans?")

faithfulness_result = faithfulness_evaluator.evaluate_response(response=response)
relevancy_result = relevancy_evaluator.evaluate_response(
    query="What is the refund window for monthly plans?", response=response
)

print("Faithful:", faithfulness_result.passing)
print("Relevant:", relevancy_result.passing)

LangChain leans on LangSmith for tracing and eval, which is a separate hosted product with a generous free tier for small projects:

import os
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "ls-..."
os.environ["LANGCHAIN_PROJECT"] = "support-rag"

# Every chain invocation is now traced automatically in LangSmith
result = qa_chain.invoke({"query": "What is the refund window for monthly plans?"})

If observability and trace-level debugging across a complex multi-step chain matters to you, LangSmith's integration with LangChain is more mature than LlamaIndex's equivalent (LlamaTrace / Arize Phoenix integration), mostly because LangChain has had more surface area (agents, tools, memory) that needed tracing for longer.

Production Concerns: Caching, Cost, and Ingestion Pipelines

For repeated ingestion runs (docs change daily, you re-index nightly), LlamaIndex's IngestionPipeline with a document store cache avoids re-embedding unchanged chunks:

from llama_index.core.ingestion import IngestionPipeline, IngestionCache
from llama_index.core.node_parser import SentenceSplitter

pipeline = IngestionPipeline(
    transformations=[SentenceSplitter(chunk_size=512), Settings.embed_model],
    cache=IngestionCache(),
)

nodes = pipeline.run(documents=documents)
pipeline.persist("./pipeline_cache")

Re-running this pipeline on an updated doc set only re-embeds chunks whose content hash changed, which matters once your embedding bill is nontrivial.

LangChain has an analogous Indexing API (langchain.indexes.SQLRecordManager) that tracks document hashes against a SQL table and skips unchanged content on re-ingestion:

from langchain.indexes import SQLRecordManager, index

record_manager = SQLRecordManager(
    "support_docs/chroma", db_url="sqlite:///record_manager_cache.sql"
)
record_manager.create_schema()

index(
    chunks,
    record_manager,
    vectorstore,
    cleanup="incremental",
    source_id_key="source",
)

Both solve the same problem (do not pay to re-embed unchanged content) with roughly equal effort. Neither has a decisive edge here.

Can You Use Both Together?

Yes, and plenty of production systems do. A common pattern: use LlamaIndex for the indexing and retrieval layer because its node parsers and query engines are more specialized, then wrap the resulting query engine as a LangChain tool so it can participate in a broader LangGraph agent.

from langchain.tools import Tool

def query_support_docs(question: str) -> str:
    response = query_engine.query(question)
    return str(response)

llamaindex_tool = Tool(
    name="support_docs_search",
    func=query_support_docs,
    description="Search support documentation using LlamaIndex's retrieval engine.",
)

This is not a hack; it is a recognized integration pattern documented by both projects. Treat "LlamaIndex vs LangChain" as a false binary when your system has both a hard retrieval problem and a hard orchestration problem.

Decision Checklist

Building a single-purpose Q&A bot over documents, with retrieval quality as the main risk: pick LlamaIndex.
Building a multi-tool agent (retrieval plus APIs plus calculators plus human approval) with branching logic: pick LangChain, likely with LangGraph.
Need advanced chunking (semantic splitting, hierarchical parent/child retrieval, auto-merging): LlamaIndex's node parsers are ahead.
Need mature tracing and step-by-step debugging across a long chain: LangChain plus LangSmith is more battle-tested.
Team already knows one framework well: that familiarity usually outweighs the marginal feature differences above. Both frameworks move fast; whichever mental model your team already has will get you to production faster than the "objectively better" choice.
Uncertain and starting from zero: build the LlamaIndex version first. It gets you a working, evaluable RAG pipeline in less code, and you can always wrap it as a tool inside LangChain later if the project grows into a multi-tool agent.

FAQ

Is LlamaIndex faster than LangChain for RAG? Query latency is dominated by the embedding model, vector store, and LLM calls, which are identical regardless of framework. Framework-level overhead is negligible in both. The real speed difference is developer speed: LlamaIndex gets a baseline RAG pipeline running in fewer lines.

Can I switch vector stores easily in both frameworks? Yes. Both use adapter patterns so swapping Chroma for Pinecone, Qdrant, or Weaviate is a constructor change, not a rewrite. Check that your chosen store has an actively maintained integration package in each framework before committing.

Does LangChain support the same chunking quality as LlamaIndex? LangChain covers the common splitters (recursive character, markdown headers, code-aware splitters) well. It does not ship LlamaIndex's semantic splitter or hierarchical node parser out of the box, though you can replicate similar behavior manually or by importing LlamaIndex just for the ingestion step.

Which one has better documentation? Both have improved significantly. LlamaIndex's docs are more retrieval-focused with worked examples for chunking strategies. LangChain's docs cover a wider surface area (agents, memory, tools) because the framework itself is broader, which can make it slower to find the exact RAG recipe you need.

Do I need LangSmith or LlamaTrace to run either framework? No, both are optional. You can run either framework with zero external tracing and just print or log responses during development. Tracing tools matter once you are debugging why a specific query returned bad context in production, not before.

Is one of these being deprecated in favor of the other? No. Both are actively maintained, widely adopted in production, and solve genuinely different core problems (retrieval-first vs orchestration-first), which is why many teams use both in the same system rather than picking a single winner.

What about frameworks like Haystack or plain custom RAG code? Haystack is a legitimate third option with strong pipeline abstractions, closer in spirit to LlamaIndex. Plain custom code (no framework) is reasonable once your RAG pipeline is simple and stable; frameworks earn their keep when you need to swap components frequently during development, not necessarily in a mature, frozen production pipeline.