Building RAG with Haystack

Haystack RAG pipelines let you wire together document stores, retrievers, and generators with a small set of composable Python classes instead of hand-rolling the plumbing yourself. If you have tried building retrieval-augmented generation from scratch, you already know the tedious parts: chunking documents, embedding them consistently, keeping the retriever and the generator in sync, and exposing the whole thing as an API that does not fall over under real traffic. Haystack solves this with a Pipeline abstraction where each step is a Component, and components snap together with typed inputs and outputs. This guide builds a complete RAG system with Haystack: document ingestion, embedding-based retrieval, prompt construction, generation, and a REST endpoint you can actually deploy.

Why Haystack for RAG instead of writing your own pipeline

Haystack is a Python framework purpose-built for search and generation pipelines, maintained by deepset. Three things make it a good fit for RAG specifically rather than a general LLM orchestration tool:

• Document stores are first-class. Haystack ships integrations for in-memory stores (great for prototyping), Elasticsearch, OpenSearch, Weaviate, Qdrant, Pinecone, and pgvector, all behind the same DocumentStore interface. Swapping the backend later does not mean rewriting your retrieval logic.
• Pipelines are declarative and inspectable. You connect components with pipeline.connect("retriever.documents", "prompt_builder.documents"), and Haystack validates the wiring at build time, not at 2am in production.
• Components are swappable independently. You can change the embedder, the retriever, or the generator without touching the rest of the graph, which matters a lot when you are benchmarking different embedding models or comparing generator providers.

The tradeoff is a small learning curve around the Pipeline and Component abstractions. Once that clicks, adding a new step (a ranker, a query rewriter, metadata filtering) is a few lines, not an afternoon of refactoring.

Setting up the environment

Install the core package plus the in-memory document store, which is enough for the whole tutorial. Swap it for a persistent store later without changing pipeline logic.

python -m venv .venv
source .venv/bin/activate
pip install haystack-ai sentence-transformers fastapi uvicorn python-dotenv

haystack-ai is the current package name for Haystack 2.x. If you find tutorials referencing farm-haystack, that is the older 1.x line with a different API; this guide uses 2.x throughout.

Set your generator API key as an environment variable so it never ends up in source control:

export OPENAI_API_KEY="your-key-here"

Haystack's OpenAIGenerator picks this up automatically, but the pipeline below is provider-agnostic: swap in any chat-completion-compatible generator component and the rest of the graph is unchanged.

Building the indexing pipeline

RAG has two pipelines that live side by side: one that indexes documents into the store, and one that answers questions using that store. Start with indexing.

from haystack import Pipeline
from haystack.components.converters import TextFileToDocument
from haystack.components.preprocessors import DocumentSplitter, DocumentCleaner
from haystack.components.embedders import SentenceTransformersDocumentEmbedder
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.document_stores.types import DuplicatePolicy

document_store = InMemoryDocumentStore()

indexing_pipeline = Pipeline()
indexing_pipeline.add_component("converter", TextFileToDocument())
indexing_pipeline.add_component("cleaner", DocumentCleaner())
indexing_pipeline.add_component(
    "splitter",
    DocumentSplitter(split_by="sentence", split_length=5, split_overlap=1),
)
indexing_pipeline.add_component(
    "embedder",
    SentenceTransformersDocumentEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"),
)
indexing_pipeline.add_component(
    "writer",
    DocumentWriter(document_store=document_store, policy=DuplicatePolicy.OVERWRITE),
)

indexing_pipeline.connect("converter.documents", "cleaner.documents")
indexing_pipeline.connect("cleaner.documents", "splitter.documents")
indexing_pipeline.connect("splitter.documents", "embedder.documents")
indexing_pipeline.connect("embedder.documents", "writer.documents")

A few decisions here matter more than they look like at first glance.

Splitting by sentence, not by fixed character count. split_by="sentence" with split_length=5 groups roughly five sentences per chunk, which keeps semantic units intact. Fixed-character splitting is faster to reason about but routinely cuts sentences in half, which hurts both embedding quality and the generator's ability to quote source text accurately.

Overlap of 1. A one-unit overlap between consecutive chunks means a fact sitting near a chunk boundary still gets fully captured in at least one chunk. Too much overlap bloats your index and slows retrieval; too little and you lose context at boundaries. Start at 1 and only increase it if you see the generator missing facts that clearly exist in your source documents.

`all-MiniLM-L6-v2` for embeddings. It is small (roughly 80MB), runs fast on CPU, and is good enough for most internal-knowledge-base use cases. If your corpus is highly technical or domain-specific (legal, medical, code), evaluate a larger model against your own queries before committing, since MiniLM's general-purpose training can miss domain nuance.

Run the indexing pipeline against a folder of text files:

import glob

file_paths = glob.glob("docs/**/*.txt", recursive=True)
indexing_pipeline.run({"converter": {"sources": file_paths}})

print(f"Indexed {document_store.count_documents()} chunks")

For PDFs, swap TextFileToDocument for PyPDFToDocument; for HTML pages, HTMLToDocument. Haystack also ships a MultiFileConverter that dispatches by file extension automatically, useful once your source corpus is mixed.

Building the retrieval-augmented generation pipeline

With documents indexed, build the pipeline that actually answers questions. This is the one your API will call on every request.

from haystack.components.embedders import SentenceTransformersTextEmbedder
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator

prompt_template = """
Answer the question using only the context below. If the context does not
contain the answer, say you don't have enough information, do not guess.

Context:
{% for document in documents %}
{{ document.content }}
{% endfor %}

Question: {{ question }}
Answer:
"""

rag_pipeline = Pipeline()
rag_pipeline.add_component(
    "text_embedder",
    SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"),
)
rag_pipeline.add_component(
    "retriever",
    InMemoryEmbeddingRetriever(document_store=document_store, top_k=5),
)
rag_pipeline.add_component("prompt_builder", PromptBuilder(template=prompt_template))
rag_pipeline.add_component("generator", OpenAIGenerator(model="gpt-4o-mini"))

rag_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
rag_pipeline.connect("retriever.documents", "prompt_builder.documents")
rag_pipeline.connect("prompt_builder.prompt", "generator.prompt")

Notice the query embedder uses the same model as the document embedder. This is not optional: if you embed documents with one model and queries with another, the vectors live in different spaces and cosine similarity becomes meaningless. Keep them locked to the same model name in a shared config value rather than hardcoding it twice.

The prompt template explicitly instructs the generator to say it lacks information rather than guess. This single instruction is the cheapest hallucination guard you can add, and skipping it is the most common reason RAG demos look great in testing and confidently make things up in production.

Query it:

question = "What is the refund window for annual subscriptions?"

result = rag_pipeline.run(
    {
        "text_embedder": {"text": question},
        "prompt_builder": {"question": question},
    }
)

print(result["generator"]["replies"][0])

Inspecting what the retriever actually found

Before trusting any answer, check what got retrieved. A wrong answer is almost always a retrieval problem, not a generation problem, so debug in that order.

rag_pipeline_debug = Pipeline()
rag_pipeline_debug.add_component("text_embedder", SentenceTransformersTextEmbedder(
    model="sentence-transformers/all-MiniLM-L6-v2"
))
rag_pipeline_debug.add_component("retriever", InMemoryEmbeddingRetriever(
    document_store=document_store, top_k=5
))
rag_pipeline_debug.connect("text_embedder.embedding", "retriever.query_embedding")

debug_result = rag_pipeline_debug.run({"text_embedder": {"text": question}})
for doc in debug_result["retriever"]["documents"]:
    print(f"score={doc.score:.3f} | {doc.content[:120]}")

If the top-scoring chunks are irrelevant, the fix is almost never a bigger generator model. Check, in order: whether the source document even contains the answer, whether your chunk size is cutting the relevant fact across two chunks, and whether top_k is too low to surface the right chunk when several similar-looking chunks compete for the top slots. Raising top_k from 5 to 8 or 10 is a cheap first experiment.

Adding metadata filtering

Real knowledge bases need scoping: only search a specific customer's documents, only search docs updated in the last quarter, only search a given product line. Haystack retrievers accept a filters argument that maps directly onto document metadata.

Attach metadata during indexing:

from haystack import Document

doc = Document(
    content="Annual subscriptions can be refunded within 30 days of purchase.",
    meta={"product": "pro-plan", "updated": "2026-03-01", "source": "billing-faq.txt"},
)
document_store.write_documents([doc])

Filter at query time:

result = rag_pipeline.run(
    {
        "text_embedder": {"text": question},
        "retriever": {
            "filters": {
                "field": "meta.product",
                "operator": "==",
                "value": "pro-plan",
            }
        },
        "prompt_builder": {"question": question},
    }
)

Haystack's filter syntax also supports AND/OR composition for multi-condition filters, which matters once you have more than one dimension to scope by (product plus recency, for example). Push scoping into filters rather than into the prompt text; filtering happens before the vector search runs, so it is both cheaper and more reliable than asking the generator to ignore irrelevant context it already received.

Wrapping the pipeline in a FastAPI endpoint

A pipeline sitting in a notebook is not a product. Expose it as an API so other services can call it.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class QueryRequest(BaseModel):
    question: str
    product_filter: str | None = None

class QueryResponse(BaseModel):
    answer: str
    sources: list[str]

@app.post("/ask", response_model=QueryResponse)
def ask(request: QueryRequest):
    pipeline_input = {
        "text_embedder": {"text": request.question},
        "prompt_builder": {"question": request.question},
    }

    if request.product_filter:
        pipeline_input["retriever"] = {
            "filters": {
                "field": "meta.product",
                "operator": "==",
                "value": request.product_filter,
            }
        }

    result = rag_pipeline.run(pipeline_input, include_outputs_from={"retriever"})

    answer = result["generator"]["replies"][0]
    sources = [doc.meta.get("source", "unknown") for doc in result["retriever"]["documents"]]

    return QueryResponse(answer=answer, sources=sources)

The include_outputs_from={"retriever"} argument is easy to miss but important: by default Haystack only returns the outputs of terminal pipeline components, so without it you would not get the retrieved documents back alongside the generated answer. Returning sources matters for user trust, since it lets you show which document backed a given answer instead of presenting the model's output as if it came from nowhere.

Run it:

uvicorn main:app --reload --port 8000

Test it:

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"question": "What is the refund window?", "product_filter": "pro-plan"}'

Moving from InMemoryDocumentStore to a persistent store

InMemoryDocumentStore is convenient for development but disappears on restart and does not scale past a small corpus. For production, swap it for a persistent store; the pipeline code above does not change, only the store initialization does. Here is the swap for pgvector, a solid default if you are already running Postgres:

from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore

document_store = PgvectorDocumentStore(
    connection_string="postgresql://user:password@localhost:5432/ragdb",
    table_name="haystack_documents",
    embedding_dimension=384,
    vector_function="cosine_similarity",
    recreate_table=False,
)

Install the integration package separately: pip install pgvector-haystack. Note embedding_dimension=384 matches all-MiniLM-L6-v2's output size; if you switch embedding models later, this number and your existing index become incompatible, and you need to reindex from scratch. Pin your embedding model choice early and treat changing it as a migration, not a config tweak.

Evaluating retrieval quality before you ship

Do not ship a RAG pipeline on vibes. Haystack includes evaluation components that score retrieval and generation against a labeled set of question-answer pairs.

from haystack.components.evaluators import (
    ContextRelevanceEvaluator,
    FaithfulnessEvaluator,
)

context_evaluator = ContextRelevanceEvaluator()
faithfulness_evaluator = FaithfulnessEvaluator()

questions = ["What is the refund window for annual subscriptions?"]
contexts = [["Annual subscriptions can be refunded within 30 days of purchase."]]
predicted_answers = ["Annual subscriptions can be refunded within 30 days."]

context_result = context_evaluator.run(questions=questions, contexts=contexts)
faithfulness_result = faithfulness_evaluator.run(
    questions=questions, contexts=contexts, predicted_answers=predicted_answers
)

print("Context relevance:", context_result["individual_scores"])
print("Faithfulness:", faithfulness_result["individual_scores"])

ContextRelevanceEvaluator scores whether the retrieved chunks actually relate to the question, separate from whether the final answer was good. FaithfulnessEvaluator scores whether the generated answer is actually supported by the retrieved context, catching hallucination even when the answer sounds plausible. Run both against a set of 20 to 50 real questions from your domain before launch, and rerun them whenever you change the chunk size, embedding model, or top_k. A change that looks like an improvement on one hand-picked example can quietly regress on the rest of your question set, and these evaluators are how you catch that before your users do.

Common mistakes to avoid

• Mismatched embedder models between indexing and querying. Covered above, but worth repeating: this is the single most common bug in Haystack RAG setups and produces retrieval results that look almost random.
• Chunking too large. A 1000-token chunk buries the relevant sentence in noise and dilutes the embedding vector. Start small (a handful of sentences) and only grow chunk size if you see the generator missing surrounding context it needs.
• No filter on retrieval when the corpus mixes tenants or topics. Without metadata filtering, a query about one customer's data can retrieve and leak chunks from another customer's documents. Add filtering before you add more documents, not after.
• Skipping the "I don't know" instruction in the prompt. Generators are optimized to be helpful, which means they will produce a confident-sounding answer even from irrelevant context unless you explicitly tell them not to.
• Not returning sources. Users trust RAG answers more, and catch bad ones faster, when they can see the document an answer came from.

FAQ

Does Haystack require OpenAI, or can I use a local model? Neither the document store nor the retriever depends on OpenAI. The OpenAIGenerator in this guide is one option among several; Haystack has generator components for other hosted providers and for local inference through Ollama or Hugging Face text-generation-inference. Swap the generator component and the rest of the pipeline is untouched.

How do I handle documents that update frequently? Reindex the changed document with DuplicatePolicy.OVERWRITE on the DocumentWriter, keyed by a stable document ID (pass id_hash_keys or set an explicit Document.id based on the source path). This replaces the old chunks for that document without duplicating them in the store.

What is the difference between InMemoryEmbeddingRetriever and a BM25 retriever? InMemoryEmbeddingRetriever does dense vector similarity search over embeddings, which is good at matching meaning even when wording differs. Haystack also ships InMemoryBM25Retriever for sparse keyword search, which is better at matching exact terms, product codes, or names. Many production pipelines run both and merge results with a DocumentJoiner, since hybrid retrieval typically outperforms either method alone.

How large a corpus can InMemoryDocumentStore realistically handle? It depends on available RAM more than document count, since every chunk and its embedding lives in process memory. It is fine for prototyping and for corpora up to a few thousand chunks. Beyond that, or for anything that needs to survive a process restart, move to a persistent backend like pgvector, Qdrant, or OpenSearch.

Can I add a reranker to improve result ordering? Yes. Add a component like SentenceTransformersDiversityRanker or a cross-encoder reranker between the retriever and the prompt builder, connecting retriever.documents to the ranker's input and the ranker's output to prompt_builder.documents. Rerankers are more expensive per query than the initial retrieval but meaningfully improve precision when top_k is high and result ordering matters.