LangChain for Document Q&A: A Complete Beginner Walkthrough

Why Document Q&A Is the First Real LangChain Project Worth Building

Every developer learning LangChain hits the same wall after the "hello world" chatbot demo: how do you make an LLM answer questions about *your* documents instead of whatever it memorized during training? That's the exact problem document Q&A solves, and it's the most practical entry point into LangChain because it forces you to learn the four skills you'll reuse in almost every serious LLM application — loading data, chunking it sensibly, embedding it into a vector store, and wiring retrieval into a prompt.

This walkthrough builds a working document Q&A system from scratch. No hand-waving, no "left as an exercise" gaps. By the end you'll have a script that loads a PDF or text file, splits it, embeds it, stores it in a vector database, retrieves relevant chunks for a question, and generates a grounded answer with citations back to the source. We'll also cover the mistakes that trip up almost everyone the first time — bad chunk sizes, forgetting metadata, and not handling the "I don't know" case.

If you're completely new to LangChain, know this upfront: it's a framework for chaining together LLM calls with external data and tools. Document Q&A (often called "retrieval-augmented generation" or RAG) is the pattern where you retrieve relevant text before asking the LLM to answer, rather than relying on the model's frozen training data. That's the entire idea. Everything below is implementation detail on top of that one concept.

Setting Up Your Environment

Start with a clean virtual environment. LangChain's ecosystem moves fast and splits functionality across multiple packages, so pin your versions once things work.

python -m venv venv
source venv/bin/activate
pip install langchain langchain-community langchain-openai langchain-text-splitters chromadb pypdf tiktoken

A quick note on package structure, because it confuses newcomers: langchain holds the core abstractions (chains, prompts, runnables), langchain-community holds third-party integrations (loaders, some vector stores), and langchain-openai holds the OpenAI-specific chat models and embeddings. If you're using a different provider, swap in langchain-anthropic or langchain-google-genai instead — the interfaces are nearly identical, only the class names and API keys differ.

Set your API key as an environment variable rather than hardcoding it:

export OPENAI_API_KEY="your-key-here"

Create a working folder with a docs/ subfolder and drop in a PDF or a few .txt files you want to query. For this walkthrough, imagine you have a company handbook PDF and want to ask questions like "How many vacation days do new hires get?"

Step 1: Loading Your Documents

LangChain's document loaders normalize different file formats into a common Document object, which has page_content (the text) and metadata (source file, page number, etc.). That metadata matters more than beginners expect — it's how you'll later show users which page an answer came from.

from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader, TextLoader

def load_documents(docs_path: str):
    """Load all PDFs and text files from a directory."""
    pdf_loader = DirectoryLoader(
        docs_path,
        glob="**/*.pdf",
        loader_cls=PyPDFLoader,
        show_progress=True,
    )
    txt_loader = DirectoryLoader(
        docs_path,
        glob="**/*.txt",
        loader_cls=TextLoader,
        show_progress=True,
    )

    documents = pdf_loader.load() + txt_loader.load()
    print(f"Loaded {len(documents)} document sections from {docs_path}")
    return documents

if __name__ == "__main__":
    docs = load_documents("./docs")
    print(docs[0].page_content[:300])
    print(docs[0].metadata)

Run this and you'll see each PDF page loaded as a separate Document with metadata like {'source': 'docs/handbook.pdf', 'page': 3}. That per-page metadata is preserved automatically by PyPDFLoader, and it's the reason you should almost always prefer format-specific loaders over dumping everything through a generic text reader.

A common beginner mistake here is loading a giant single document and treating it as one blob. Don't do that — a 40-page PDF is 40 Document objects at this stage, each representing one page. That's still too coarse for retrieval, which is exactly what the next step fixes.

Step 2: Splitting Text Into Retrievable Chunks

Feeding entire pages into an embedding model wastes context and hurts retrieval precision — if a page covers three unrelated policies, a question about one policy will pull in noise from the other two. You need to split documents into smaller, semantically coherent chunks.

RecursiveCharacterTextSplitter is the default choice for a reason: it tries to split on paragraph breaks first, then sentences, then words, only falling back to a hard character cut as a last resort. This keeps related sentences together far better than a naive fixed-length split.

from langchain_text_splitters import RecursiveCharacterTextSplitter

def split_documents(documents, chunk_size=1000, chunk_overlap=150):
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        separators=["\n\n", "\n", ". ", " ", ""],
    )
    chunks = splitter.split_documents(documents)
    print(f"Split {len(documents)} documents into {len(chunks)} chunks")
    return chunks

chunks = split_documents(docs)
print(chunks[5].page_content)
print(chunks[5].metadata)

Two parameters decide almost everything about retrieval quality:

• chunk_size — how many characters per chunk. Too small (under 300) and chunks lose context, causing the LLM to answer confidently from a fragment that's missing the qualifying sentence. Too large (over 2000) and you dilute the embedding, making semantically distinct passages look similar to the vector search. Start at 800-1200 characters for prose-heavy documents like handbooks or manuals.
• chunk_overlap — how many characters repeat between consecutive chunks. This prevents a sentence from being sliced exactly at a chunk boundary and losing meaning on both sides. 10-20% of chunk_size is a reasonable default.

Notice that split_documents preserves and copies the source metadata onto every chunk. That's not automatic magic — it's a deliberate design in LangChain's splitters, and it's why you should always split Document objects rather than splitting raw strings and rebuilding metadata yourself later.

Step 3: Embedding and Storing in a Vector Database

Embeddings convert text into numeric vectors positioned so that semantically similar text ends up close together in vector space. A vector store lets you search "what's near this query vector" efficiently. For a beginner walkthrough, Chroma is the right choice — it runs locally, persists to disk, and needs zero infrastructure setup.

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

def build_vector_store(chunks, persist_directory="./chroma_db"):
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=persist_directory,
    )
    return vector_store

vector_store = build_vector_store(chunks)
print("Vector store built and persisted to ./chroma_db")

The first time you run this, LangChain sends every chunk to the embeddings API and stores the resulting vectors alongside the text and metadata in a local Chroma database. This step costs API calls proportional to your document size, so persist the database and only rebuild it when your source documents change — don't re-embed on every script run.

To reload an existing store without re-embedding:

def load_vector_store(persist_directory="./chroma_db"):
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    return Chroma(
        persist_directory=persist_directory,
        embedding_function=embeddings,
    )

Test the retrieval mechanism directly before wiring in the LLM, so you can debug retrieval quality independently of generation quality:

results = vector_store.similarity_search("How many vacation days do new hires get?", k=3)
for i, doc in enumerate(results):
    print(f"--- Result {i+1} (page {doc.metadata.get('page')}) ---")
    print(doc.page_content[:200])

If the retrieved chunks don't actually contain the answer, no amount of prompt engineering downstream will fix it — go back and adjust chunk size or check that the right documents were loaded in step 1.

Step 4: Building the Retrieval Chain With LCEL

Now connect retrieval to generation using LangChain Expression Language (LCEL), the pipe-based syntax that replaced the older RetrievalQA chain class. LCEL chains are easier to debug and stream than the legacy chain classes, and they're what current LangChain documentation and courses teach as the standard approach.

The core idea: take the user's question, retrieve relevant chunks, stuff them into a prompt template alongside the question, and send that to the LLM.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

retriever = vector_store.as_retriever(search_kwargs={"k": 4})

prompt = ChatPromptTemplate.from_template("""
You are a helpful assistant answering questions using only the provided context.
If the answer isn't in the context, say "I don't have enough information to answer that"
instead of guessing.

Context:
{context}

Question: {question}

Answer:
""")

def format_docs(docs):
    return "\n\n".join(
        f"[Source: {d.metadata.get('source', 'unknown')}, page {d.metadata.get('page', '?')}]\n{d.page_content}"
        for d in docs
    )

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

answer = rag_chain.invoke("How many vacation days do new hires get?")
print(answer)

Walk through what that pipeline actually does, because the | syntax hides a lot of structure:

1. The dictionary {"context": retriever | format_docs, "question": RunnablePassthrough()} runs two things in parallel: it sends the input question to the retriever (which returns chunks, then formats them into a string), and it passes the raw question through unchanged.
2. Both outputs land in the prompt template's {context} and {question} placeholders.
3. The filled prompt goes to the LLM.
4. StrOutputParser() extracts the plain text string from the LLM's response object instead of returning the full message object.

That explicit "say you don't know instead of guessing" instruction in the prompt is not optional boilerplate — it's the single highest-leverage line in this entire walkthrough. Without it, LLMs will confidently fabricate plausible-sounding answers when the retrieved context doesn't actually contain the information, and that failure mode is much harder to catch than an outright error.

Step 5: Returning Sources Alongside Answers

Production Q&A systems almost always need to show users *where* an answer came from, both for trust and for verification. Extend the chain to return the source documents alongside the generated answer instead of just the final string.

from langchain_core.runnables import RunnableParallel

rag_chain_with_sources = RunnableParallel(
    {"context": retriever, "question": RunnablePassthrough()}
).assign(answer=(
    {"context": lambda x: format_docs(x["context"]), "question": lambda x: x["question"]}
    | prompt
    | llm
    | StrOutputParser()
))

result = rag_chain_with_sources.invoke("How many vacation days do new hires get?")

print("Answer:", result["answer"])
print("\nSources used:")
for doc in result["context"]:
    print(f"- {doc.metadata.get('source')} (page {doc.metadata.get('page')})")

RunnableParallel runs the retriever once and keeps its raw output available under the context key, while .assign() adds the answer key computed from that same context. This avoids retrieving twice, and it gives you a structured result object you can render in a UI with citations, rather than a plain string the user has to trust blindly.

Choosing Between Chroma, FAISS, and Hosted Vector Databases

Chroma is the right default for learning and for small-to-medium projects, but it's worth understanding the tradeoffs before you commit to it in a real product, because swapping vector stores later means re-touching every place you build or query the store.

• Chroma — embedded, file-based, zero setup. Great for prototypes, single-machine apps, and anything under a few hundred thousand chunks. Persistence is just a folder on disk, which also makes it trivial to back up or ship inside a Docker image.
• FAISS — Facebook's similarity search library, wrapped by langchain_community.vectorstores.FAISS. Faster for pure in-memory search at larger scale, but it doesn't handle metadata filtering as gracefully as Chroma and has no built-in persistence server — you manage the index file yourself.
• Hosted options (Pinecone, Weaviate, Qdrant, pgvector on Postgres) — worth it once you need multi-user concurrent access, horizontal scaling, or you're already running Postgres and want to avoid adding new infrastructure (pgvector is a strong choice in that case). The LangChain interface barely changes: you still call .from_documents() and .as_retriever(), you just swap the import and add connection credentials.

The practical advice: build and validate your Q&A logic with Chroma first, exactly as this walkthrough does. Only migrate to a hosted vector database once you have a concrete scaling reason — more than one process needing to query the same store concurrently, a document corpus in the millions of chunks, or a need for role-based access control on who can retrieve what. Migrating the vector store later is a one-file change in the code you've already written; it is not a reason to over-engineer the prototype.

Adding Conversational Memory for Follow-Up Questions

The chain built so far treats every question as a fresh, standalone query. Ask "How many vacation days do new hires get?" and then follow up with "What about after five years?" and the second question will fail, because the retriever has no idea "what about" refers to vacation days — it'll search the vector store for a query that's missing its own subject.

Fixing this requires rewriting the follow-up question into a standalone one before it hits the retriever, using the conversation history as context:

from langchain_core.prompts import MessagesPlaceholder
from langchain_core.messages import HumanMessage, AIMessage

condense_prompt = ChatPromptTemplate.from_messages([
    ("system", "Rewrite the latest user question as a standalone question, "
               "using the chat history for context. Do not answer it."),
    MessagesPlaceholder("chat_history"),
    ("human", "{question}"),
])

condense_chain = condense_prompt | llm | StrOutputParser()

def conversational_qa(question, chat_history):
    if chat_history:
        standalone_question = condense_chain.invoke({
            "question": question,
            "chat_history": chat_history,
        })
    else:
        standalone_question = question

    answer = rag_chain.invoke(standalone_question)

    chat_history.append(HumanMessage(content=question))
    chat_history.append(AIMessage(content=answer))
    return answer, chat_history


history = []
answer, history = conversational_qa("How many vacation days do new hires get?", history)
print(answer)

answer, history = conversational_qa("What about after five years?", history)
print(answer)

This two-step pattern — condense, then retrieve-and-answer — is the same approach behind LangChain's older ConversationalRetrievalChain, just expressed explicitly with LCEL so you can see and modify every step. Keeping it explicit like this also makes it much easier to debug: if a follow-up question gets a bad answer, you can print standalone_question and immediately see whether the condensing step or the retrieval step is at fault.

Streaming Answers for a Better User Experience

Waiting several seconds for a full answer to appear feels sluggish, especially once your context window (and therefore your response length) grows. LangChain's runnables support streaming out of the box — you don't need a different chain, just a different invocation method.

for chunk in rag_chain.stream("How many vacation days do new hires get?"):
    print(chunk, end="", flush=True)

Because rag_chain is built from composable runnables ending in StrOutputParser(), calling .stream() instead of .invoke() yields the answer token-by-token as the LLM generates it, rather than blocking until the entire response is ready. If you're building a web front end, the same chain works behind a FastAPI endpoint using StreamingResponse, so the retrieval and prompt logic you've already written doesn't need to change at all when you move from a CLI script to a real application — only the transport layer around it does.

Evaluating Whether Your Q&A System Actually Works

It's tempting to eyeball a handful of answers and call the system done, but that doesn't scale past the demo stage, and it won't catch regressions when you tweak chunk size or swap models. Build a small evaluation set early:

eval_questions = [
    {"question": "How many vacation days do new hires get?", "expected_keywords": ["10", "days", "vacation"]},
    {"question": "What is the process for requesting parental leave?", "expected_keywords": ["parental", "leave", "HR"]},
]

def run_eval(chain, eval_set):
    passed = 0
    for item in eval_set:
        answer = chain.invoke(item["question"])
        hit = any(kw.lower() in answer.lower() for kw in item["expected_keywords"])
        status = "PASS" if hit else "FAIL"
        if hit:
            passed += 1
        print(f"[{status}] {item['question']}\n  -> {answer[:150]}")
    print(f"\n{passed}/{len(eval_set)} passed")

run_eval(rag_chain, eval_questions)

This keyword-based check is intentionally crude — it won't catch subtle factual errors — but it's enough to flag obvious regressions automatically every time you change chunking, embeddings, or the prompt. As your project matures, replace it with an LLM-as-judge evaluator that scores answers for faithfulness to the retrieved context, which catches hallucination even when the surface-level keywords happen to match.

Step 6: Wrapping It Into a Reusable Q&A Script

Pull everything into one script so you can run it against any folder of documents from the command line.

import sys
from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser


def build_qa_system(docs_path="./docs", persist_directory="./chroma_db"):
    pdf_loader = DirectoryLoader(docs_path, glob="**/*.pdf", loader_cls=PyPDFLoader)
    txt_loader = DirectoryLoader(docs_path, glob="**/*.txt", loader_cls=TextLoader)
    documents = pdf_loader.load() + txt_loader.load()

    splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)
    chunks = splitter.split_documents(documents)

    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    vector_store = Chroma.from_documents(chunks, embeddings, persist_directory=persist_directory)
    retriever = vector_store.as_retriever(search_kwargs={"k": 4})

    llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

    prompt = ChatPromptTemplate.from_template("""
Answer using only the context below. If the answer isn't there, say you don't know.

Context:
{context}

Question: {question}
Answer:
""")

    def format_docs(docs):
        return "\n\n".join(d.page_content for d in docs)

    chain = (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )
    return chain


if __name__ == "__main__":
    qa_chain = build_qa_system()
    print("Document Q&A ready. Type 'exit' to quit.\n")
    while True:
        question = input("Ask a question: ")
        if question.strip().lower() == "exit":
            break
        response = qa_chain.invoke(question)
        print(f"\n{response}\n")

Run it with python qa_system.py and you get an interactive loop that answers questions grounded in your own documents. This is a genuinely usable prototype — the kind of thing you could point at internal wikis, policy documents, or product manuals today.

Common Pitfalls and How to Debug Them

A few failure patterns show up constantly for people building their first document Q&A system:

• Retrieval returns irrelevant chunks. Usually a chunking problem, not an embedding problem. Print similarity_search results directly (as shown in step 3) before blaming the LLM. Try smaller chunk sizes or increasing k in search_kwargs.
• The LLM answers confidently even when it shouldn't know. This means your prompt isn't strict enough about staying within context. Strengthen the "don't know" instruction, or add a temperature=0 setting if you haven't already — deterministic output reduces creative guessing.
• Rebuilding the vector store on every run. Wastes money and time. Check persist_directory for existing data before calling Chroma.from_documents again, and only rebuild when source files change.
• Ignoring metadata. If you don't pass metadata through consistently, you lose the ability to cite sources, which is often the actual deliverable users care about most.
• Using one giant chunk size for structurally different documents. A dense legal contract and a casual FAQ page shouldn't use the same chunk_size. Tune per document type when your corpus is mixed.

Where to Go From Here

This walkthrough covers the linear, single-pass RAG pattern: load, split, embed, retrieve, generate. It's the foundation, but real systems layer more on top — re-ranking retrieved chunks before generation, hybrid search that combines keyword and vector search, multi-query retrieval that rephrases the user's question several ways to widen recall, and conversational memory so follow-up questions understand context from earlier in the chat. Each of those is a small, composable addition to the chain you just built, not a rewrite.

If you want to go deeper — building agentic document assistants, adding conversational memory, evaluating retrieval quality systematically, and deploying a production-ready Q&A API — that's exactly the ground we cover in the LangChain Tutorial 2026 course on teachyou.ai, with hands-on projects that pick up right where this walkthrough leaves off.