Building a RAG Application with Dify

Dify RAG pipelines let you ship a document-grounded chatbot without stitching together a vector database, an embedding service, and an orchestration layer by hand. Dify is an open source LLM application platform with a visual workflow builder, a built-in knowledge base module, and a one-click API layer, so a working retrieval-augmented generation app can go from zero to a callable endpoint in an afternoon. This guide walks through the full build: standing up Dify, ingesting documents into a knowledge base, wiring a retrieval-aware chat workflow, tuning the retrieval settings, and exposing the result as an API you can call from any frontend.

By the end you will have a running Dify RAG app that answers questions strictly from a document set you control, with citations back to the source chunks and a debuggable pipeline you can iterate on without touching a line of Python.

What Dify Brings to a RAG Stack

A typical RAG stack has five moving parts: a document loader, a chunker, an embedding model, a vector store, and an orchestrator that stitches retrieved chunks into a prompt before calling an LLM. Building that from scratch means picking a chunking strategy, choosing an embedding provider, standing up a vector database, and writing the retrieval-then-generate glue code yourself.

Dify collapses all five into one product surface:

• Knowledge Base module: handles document upload, chunking, embedding, and indexing in one screen. Supports PDFs, Word docs, Markdown, plain text, CSV, and web page imports.
• Built-in vector store: ships with Weaviate by default in self-hosted deployments, and supports swapping in Qdrant, Milvus, PGVector, or Elasticsearch through environment configuration.
• Retrieval settings UI: lets you switch between vector search, full-text search, and hybrid search, and tune top-k and score thresholds without redeploying anything.
• Workflow / Chatflow builder: a node-based canvas where a "Knowledge Retrieval" node feeds context straight into an "LLM" node, with conditional branches, variable aggregation, and code nodes available if you need custom logic.
• One-click API and SDKs: every app you build gets a REST endpoint, an API key, and a web widget out of the box.

This matters because the retrieval quality problem in RAG, not the plumbing, is where most of the real engineering time should go. Dify RAG setups free you to spend that time on chunk size, retrieval mode, and prompt structure instead of re-implementing a vector store client.

Setting Up Dify Locally

Dify ships as a Docker Compose stack. This is the fastest path to a working instance for development.

git clone https://github.com/langgenius/dify.git
cd dify/docker
cp .env.example .env
docker compose up -d

Give the stack a minute to initialize, then check that every container reports healthy:

docker compose ps

Once the containers are up, open the console in a browser:

http://localhost/install

The first visit prompts you to create an admin account. After that, you land in the Dify workspace, which is where every step below happens.

If you would rather skip infrastructure entirely, Dify also offers a hosted cloud version with the same console. The workflow described in this article is identical either way; only the deployment step changes.

Connecting a Model Provider

Before building anything, add at least one LLM provider and one embedding model under Settings > Model Provider. Dify supports OpenAI, Anthropic, Azure OpenAI, and a long list of open source and self-hosted options through Ollama or vLLM. For a RAG app you need two model slots filled in:

• A chat/completion model for the generation step.
• An embedding model for indexing and querying the knowledge base.

Paste in your API key for each provider under its settings card. Dify validates the key immediately, so a typo shows up as a red error rather than a silent failure later in the pipeline.

Creating a Knowledge Base

Navigate to Knowledge Base > Create Knowledge Base and choose Import from file. Upload the documents you want the app to reason over. For this walkthrough, imagine a support knowledge base built from product docs, a pricing FAQ, and a troubleshooting guide exported as Markdown and PDF.

Dify walks you through three configuration screens before indexing starts.

Chunking strategy. Pick between General (fixed-length chunks with overlap) and Parent-Child (Dify indexes small child chunks for precise retrieval but returns the larger parent chunk as context, which keeps answers coherent even when the matching snippet is short). For support docs and FAQs, start with General chunking at a chunk size of 500 tokens and an overlap of 50 tokens. For long-form technical guides, Parent-Child chunking usually gives better answers because it avoids truncating a procedure mid-step.

Index method. Choose High Quality, which uses your configured embedding model to build vector embeddings, over Economy, which uses keyword-based indexing only. High Quality is what enables semantic search, and it is required for a real RAG pipeline.

Retrieval setting. This is set at the knowledge base level as a default, and you can still override it per app later. The three options are:

• Vector Search: pure semantic similarity, good for conceptual questions.
• Full-Text Search: keyword/BM25 matching, good for exact terms, error codes, or product names.
• Hybrid Search: runs both and merges results with a reranker, which is the best default for most support and documentation use cases.

Select Hybrid Search and confirm. Dify starts chunking and embedding the uploaded files, and you can watch per-document indexing status update in real time on the knowledge base overview page.

Inspecting and Fixing Chunks

Once indexing finishes, click into any document to see its generated chunks. This step is easy to skip and is exactly where most RAG quality problems originate. Look for:

• Chunks that cut a table or code block in half.
• Chunks that are mostly whitespace or navigation boilerplate from a PDF export.
• Duplicate chunks from repeated headers or footers.

Dify lets you manually edit, delete, or re-chunk individual segments from this screen, so a bad PDF export does not require re-uploading the whole file.

Building the Retrieval-Augmented Chatflow

With the knowledge base indexed, go to Studio > Create App and choose Chatflow (the node-based builder, as opposed to the simpler Chatbot template, gives you full control over the retrieval step and is worth the extra setup for anything beyond a toy demo).

A minimal Dify RAG chatflow has four nodes:

1. Start: captures the user's incoming message as a variable, typically sys.query.
2. Knowledge Retrieval: takes sys.query as input, searches the knowledge base you created, and returns the top matching chunks as a list variable.
3. LLM: receives a system prompt plus the retrieved chunks and the user query, and generates the answer.
4. Answer: streams the LLM node's output back to the user.

Drag these onto the canvas and connect them in order. On the Knowledge Retrieval node, select the knowledge base you built and set:

• Top K: 4 to 6 for most FAQ-style bases; push higher for dense technical docs.
• Score threshold: 0.5 as a starting point; raise it if you see irrelevant chunks slipping in, lower it if the app says "I don't know" too often.

On the LLM node, write a system prompt that explicitly instructs the model to answer only from retrieved context:

You are a support assistant for a SaaS product.
Answer the user's question using ONLY the context below.
If the context does not contain the answer, say you don't know
and suggest contacting support instead of guessing.

Context:
{{#context#}}

Question:
{{#sys.query#}}

The {{#context#}} variable is the automatically populated output of the Knowledge Retrieval node; Dify wires this reference for you when you insert a variable inside the prompt editor. This single instruction, answer only from context, is the difference between a RAG app and a chatbot that happens to have a knowledge base attached but still hallucinates when retrieval comes up empty.

Adding a Fallback Branch

A more robust chatflow adds an IF/ELSE node right after Knowledge Retrieval, checking whether any chunks were returned above the score threshold. If nothing qualifies, route to a separate LLM node (or a fixed Answer node) that responds with a clear "I couldn't find that in the docs" message instead of letting the main LLM node improvise. This one branch removes most of the hallucination risk that shows up when a user asks something genuinely outside the knowledge base.

Testing and Tuning Retrieval

Use the Preview panel on the right side of the Studio canvas to run test queries without publishing anything. For each test query, Dify shows you exactly which chunks were retrieved and their similarity scores, which turns retrieval tuning into a fast, visible loop rather than guesswork.

Run through a checklist of query types:

• A direct factual question that should hit one chunk cleanly.
• A question phrased differently from the source document's wording, to test whether semantic search is doing real work.
• A question using an exact error code or product name, to confirm the full-text half of hybrid search is contributing.
• A question genuinely outside the knowledge base, to confirm the fallback branch fires.

If retrieval quality is weak on paraphrased questions, the embedding model or chunk size is usually the culprit; try a smaller chunk size or a stronger embedding model. If exact-match queries fail, check that Hybrid Search (not pure Vector Search) is active on the knowledge base.

Swapping in an External Vector Database

The bundled vector store is fine for development and small knowledge bases, but production deployments with large document sets or strict infrastructure requirements often need an external vector database. Dify supports this through the .env file used by the Docker Compose stack.

To point Dify at an external Qdrant instance, edit the vector store block in docker/.env:

VECTOR_STORE=qdrant
QDRANT_URL=https://your-qdrant-host:6333
QDRANT_API_KEY=your-qdrant-api-key

Restart the stack for the change to take effect:

docker compose down
docker compose up -d

Existing knowledge bases need to be re-indexed after a vector store swap, since embeddings are not portable between store backends. Plan this change before you have a large production knowledge base, not after.

Publishing and Calling the API

Once the chatflow behaves the way you want in Preview, click Publish in the top right of the Studio canvas. This makes the app live and generates:

• A hosted web app URL you can share directly.
• An embeddable chat widget snippet for dropping into an existing site.
• A REST API with its own API key, under API Access.

The REST API is the integration point for anything you build outside Dify's own console. A basic call looks like this:

curl -X POST 'http://localhost/v1/chat-messages' \
  -H 'Authorization: Bearer app-YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "inputs": {},
    "query": "How do I reset my API key?",
    "response_mode": "streaming",
    "user": "user-123"
  }'

response_mode accepts streaming for server-sent events or blocking for a single JSON response, which is easier to work with when calling the API from a backend job rather than a live chat UI. The response payload includes the generated answer plus a retriever_resources array listing which knowledge base chunks were used, so you can render citations in your own frontend without any extra retrieval logic.

A minimal Python client for the blocking mode:

import requests

url = "http://localhost/v1/chat-messages"
headers = {
    "Authorization": "Bearer app-YOUR_API_KEY",
    "Content-Type": "application/json",
}
payload = {
    "inputs": {},
    "query": "How do I reset my API key?",
    "response_mode": "blocking",
    "user": "user-123",
}

response = requests.post(url, headers=headers, json=payload, timeout=30)
data = response.json()
print(data["answer"])
for chunk in data.get("metadata", {}).get("retriever_resources", []):
    print(chunk["document_name"], chunk["score"])

This is enough to wire the Dify RAG endpoint into an existing product, whether that is a support widget, an internal Slack bot, or a backend service that batch-answers a queue of tickets.

Monitoring and Iterating in Production

Dify's Logs & Annotations tab records every conversation the published app handles, including the retrieved chunks and the final answer for each turn. Two habits keep a RAG app healthy after launch:

• Review low-confidence answers weekly. Filter logs for conversations where the fallback branch fired or the LLM node's response includes hedging language. These are the queries your knowledge base does not cover yet, and they are a direct signal for what to add next.
• Use Annotations to correct bad answers directly. When an answer is wrong but retrieval was fine, you can annotate the correct response in the log view. Annotated answers get served directly for matching future queries, which fixes urgent issues immediately while you work out why the underlying prompt or chunking produced the wrong answer.

Re-index documents whenever the source material changes; Dify does not watch source files for changes automatically, so a content update needs a manual re-sync in the Knowledge Base tab or an API call to the document update endpoint if you want to automate it.

FAQ

Does Dify require coding to build a RAG app? No. The Chatflow builder, knowledge base ingestion, and retrieval configuration are all handled through the visual console. Code nodes are available inside a workflow if you need custom logic, but a complete RAG pipeline can be built and published without writing any code.

Can I use my own vector database with Dify? Yes. Self-hosted Dify supports Weaviate (the default), Qdrant, Milvus, PGVector, and Elasticsearch, configured through environment variables in the Docker Compose setup. Switching stores requires re-indexing existing knowledge bases.

What is the difference between Vector Search, Full-Text Search, and Hybrid Search in Dify? Vector Search matches on semantic similarity using embeddings, Full-Text Search matches on exact keywords using BM25-style scoring, and Hybrid Search runs both and merges the results with a reranker. Hybrid Search is the recommended default for most document sets because it handles both paraphrased questions and exact-term lookups.

How do I stop the model from answering questions outside my knowledge base? Combine an explicit system prompt instruction (answer only from the provided context) with an IF/ELSE branch after the Knowledge Retrieval node that checks the retrieval score threshold and routes low-confidence queries to a fixed fallback response instead of the main LLM node.

Can a Dify RAG app be embedded in an existing website? Yes. Publishing an app generates an embeddable chat widget snippet alongside the hosted web app URL and the REST API key, so you can drop the widget into an existing site without building a custom chat UI.

How does Parent-Child chunking differ from General chunking? General chunking splits documents into fixed-size chunks with overlap and indexes and returns the same chunk. Parent-Child chunking indexes smaller child chunks for precise matching but returns the larger parent chunk as context, which tends to produce more coherent answers for long-form or procedural documents.