RAG

Building a RAG Chatbot with Streaming Responses

Ira Menon · May 14, 2026 · 15 min read

The first time you demo a RAG chatbot to a stakeholder, the question is never "is the answer correct." It's "why did it just sit there for four seconds before saying anything." Users forgive a slightly imperfect answer far more easily than they forgive a frozen screen. That's the whole reason streaming exists, and it's why RAG streaming responses have quietly become table stakes for any chatbot built on top of a knowledge base. If you've already built a basic retrieval-augmented generation pipeline and gotten it working in a Jupyter notebook, this is the part where you turn it into something that feels alive in a browser. I'm going to walk through the architecture, the retrieval layer, the streaming mechanics on both the server and client, and the failure modes that only show up once real users start typing questions you didn't anticipate.

Why streaming matters more in RAG than in plain chat

With a plain LLM chat wrapper, streaming is a nice-to-have. With RAG, it's closer to a necessity, because RAG adds latency on the front end that users don't see. Before a single token of the answer gets generated, your system has to embed the query, search a vector store, maybe rerank the candidates, and then stuff the retrieved chunks into a prompt. That retrieval step alone can eat 300ms to 2 seconds depending on your index size and whether you're doing a single vector search or a hybrid search with reranking.

If you wait for the full generation to complete before showing anything, the user is staring at a spinner through retrieval time plus generation time. Stack those together and you're regularly north of five seconds for a moderately long answer. Streaming doesn't make retrieval faster, but it changes what the user perceives: the moment the first token of the answer appears, the wait "ends" psychologically even though the model is still writing. Getting this right is the difference between a chatbot that feels like a search engine and one that feels like a colleague typing you a reply.

The two-stage latency problem

It helps to think of RAG streaming as having two distinct phases that need different UI treatment:

Retrieval phase — embedding the query, searching the vector database, optionally reranking, and assembling context. This phase produces no tokens, so there's nothing to stream yet.
Generation phase — the LLM call itself, which is where token-by-token streaming actually happens.

A common mistake is treating these as one blob and just showing a generic "Thinking..." spinner for the whole thing. A better pattern is to surface intermediate status during retrieval ("Searching your documents...", "Found 6 relevant passages...") and then switch to actual token streaming once generation starts. This is sometimes called "status streaming" and it's cheap to build because you're not streaming tokens yet, just short-lived state updates over the same connection you'll use for the real stream.

async def rag_stream(query: str):
    yield {"type": "status", "message": "Searching documents..."}
    chunks = await retrieve(query)

    yield {"type": "status", "message": f"Found {len(chunks)} relevant passages"}
    context = build_context(chunks)

    yield {"type": "status", "message": "Generating answer..."}
    async for token in generate_answer(query, context):
        yield {"type": "token", "content": token}

    yield {"type": "done", "sources": [c.metadata for c in chunks]}

This single generator function is the backbone of the whole system. Everything downstream — the API route, the SSE handler, the frontend renderer — just consumes this stream of typed events.

Building the retrieval layer

Before you can stream anything meaningful, the retrieval step needs to be fast and predictable, because a slow retrieval phase makes the "Searching..." status linger awkwardly. A reasonable retrieval pipeline for a document-based chatbot looks like this:

Embed the incoming query with the same embedding model used to index your documents (mismatched embedding models is the single most common bug in RAG systems people ship).
Run a vector similarity search against your store, pulling back a generous candidate set — 15 to 25 chunks rather than the final 3 to 5 you'll actually use.
Rerank the candidates with a cross-encoder or a lightweight reranking API call to push the truly relevant chunks to the top.
Truncate to the top-k chunks that fit your context budget, deduplicating near-identical chunks from the same source document.
Assemble the final context block, tagging each chunk with a source identifier so you can cite it later.

from qdrant_client import QdrantClient

client = QdrantClient(url=QDRANT_URL)

async def retrieve(query: str, top_k: int = 5):
    query_vector = await embed_text(query)

    hits = client.query_points(
        collection_name="docs",
        query=query_vector,
        limit=20,
    ).points

    reranked = rerank(query, [h.payload["text"] for h in hits])
    top_chunks = reranked[:top_k]

    return [
        {"text": chunk.text, "source": chunk.metadata["source"], "score": chunk.score}
        for chunk in top_chunks
    ]

Notice that reranking happens after the initial vector search, not instead of it. Vector search is optimized for recall — casting a wide net cheaply — while reranking is optimized for precision on a small candidate set. Doing both in sequence consistently produces better final chunks than either alone, and it's a cheap addition since reranking 20 short passages is fast compared to embedding an entire corpus.

Server-side streaming with Server-Sent Events

For a chatbot, Server-Sent Events (SSE) are usually the right transport, not WebSockets. SSE is simpler, works over plain HTTP, survives through most proxies and load balancers without special configuration, and gives you a natural one-directional stream from server to client, which is exactly the shape of a chat response. WebSockets earn their complexity when you need bidirectional real-time communication — think collaborative editing or live cursors — but for "user sends one message, server streams back one answer," SSE is the leaner choice.

Here's a FastAPI endpoint that wires the RAG generator into an SSE response:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import json

app = FastAPI()

@app.post("/chat/stream")
async def chat_stream(request: ChatRequest):
    async def event_generator():
        async for event in rag_stream(request.query):
            yield f"data: {json.dumps(event)}\n\n"

    return StreamingResponse(
        event_generator(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no",
        },
    )

That X-Accel-Buffering: no header is easy to miss and it's the reason so many people get streaming working locally and then watch it arrive as one giant chunk in production. Nginx buffers proxied responses by default, which silently defeats streaming even though your server is emitting tokens correctly. If you're behind Nginx, you need that header, and you also need proxy_buffering off set in the relevant location block in your Nginx config, since the response header alone doesn't always override a strict reverse proxy setup.

If you're on Node instead of Python, the same shape holds with Express or a raw HTTP handler:

app.post('/chat/stream', async (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');

  for await (const event of ragStream(req.body.query)) {
    res.write(`data: ${JSON.stringify(event)}\n\n`);
  }
  res.end();
});

Streaming tokens from the LLM provider

Most model providers give you a streaming option on the chat completion endpoint that yields partial deltas instead of the full response. Wrapping this into your own generator keeps your RAG logic decoupled from whichever provider you're calling, which matters a lot if you ever want to swap models or run an A/B test between two providers.

async def generate_answer(query: str, context: str):
    prompt = f"""Answer the question using only the context below.
If the context doesn't contain the answer, say so plainly.

Context:
{context}

Question: {query}"""

    stream = await llm_client.chat.completions.create(
        model="claude-sonnet-5",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )

    async for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            yield delta

A detail worth calling out: the instruction "say so plainly" when the context doesn't contain an answer is doing real work here. Without an explicit instruction to admit uncertainty, models under RAG prompting will frequently blend retrieved context with their own parametric knowledge and produce an answer that sounds grounded but isn't. Being blunt in the prompt about what to do when retrieval comes up empty is one of the highest-leverage lines you can add.

The frontend: consuming the stream

On the client, you need to parse the SSE stream and update state token by token without triggering a full re-render on every single token, or you'll tank performance on longer answers. Using the native EventSource API is fine for GET requests, but since chat is almost always a POST with a body, most people reach for the Fetch API with a manual stream reader instead:

async function streamChat(query, onEvent) {
  const response = await fetch('/chat/stream', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ query }),
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = '';

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split('\n\n');
    buffer = lines.pop();

    for (const line of lines) {
      if (line.startsWith('data: ')) {
        const event = JSON.parse(line.slice(6));
        onEvent(event);
      }
    }
  }
}

The buffering logic here matters more than it looks like it should. Network chunks don't respect your message boundaries — a single data: {...}\n\n block can arrive split across two reads, or two full events can arrive in one read. Keeping a running buffer and only processing complete \n\n-terminated segments is what keeps JSON.parse from throwing on a half-received event. This is the bug that shows up intermittently in production and never in local testing, because local network chunks tend to arrive whole.

On the React side, appending tokens to a single string in state and batching renders keeps things smooth:

function useChatStream() {
  const [answer, setAnswer] = useState('');
  const [status, setStatus] = useState('');
  const [sources, setSources] = useState([]);

  const ask = async (query) => {
    setAnswer('');
    setStatus('');
    await streamChat(query, (event) => {
      if (event.type === 'status') setStatus(event.message);
      if (event.type === 'token') setAnswer((prev) => prev + event.content);
      if (event.type === 'done') {
        setSources(event.sources);
        setStatus('');
      }
    });
  };

  return { answer, status, sources, ask };
}

React batches these setAnswer calls reasonably well by default in modern versions, but if you notice jank on long answers, throttling the append to every 2-3 tokens instead of every single one is a simple fix that's invisible to the user and meaningfully cheaper to render.

Handling citations without breaking the stream

One thing that's genuinely tricky about RAG streaming responses compared to plain chat streaming is citations. Users trust a RAG chatbot more when they can see which document backed up a claim, but you don't want to wait until the very end to reveal sources, and you don't want the model inventing citation markers mid-generation that don't map to anything real.

The cleanest pattern I've found is to keep citation numbering entirely separate from generation. Assign each retrieved chunk a stable index before the prompt is built, instruct the model to reference chunks by that index inline ([1], [2]), and then resolve those markers to actual source metadata on the frontend once the done event arrives with the full source list. This way the model never has to "know" about document titles or URLs during generation — it just emits small integers, and your UI does the mapping. It also means citation rendering never blocks token streaming, since resolution happens after the stream closes.

def build_context(chunks):
    return "\n\n".join(
        f"[{i+1}] {chunk['text']}" for i, chunk in enumerate(chunks)
    )

Letting users cancel a response in flight

Once your chatbot streams, users expect the same affordance every streaming chat product trains them to expect: a stop button. This sounds trivial until you realize cancellation has to propagate all the way from a client click down through your API layer to the actual LLM provider connection, or you'll cancel the UI update while the backend keeps generating (and billing) in the background.

The cleanest way to do this on the frontend is with an AbortController, wired into the same fetch call that opened the stream:

let controller = null;

async function streamChat(query, onEvent) {
  controller = new AbortController();

  const response = await fetch('/chat/stream', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ query }),
    signal: controller.signal,
  });

  const reader = response.body.getReader();
  // ...same read loop as before
}

function stopStreaming() {
  controller?.abort();
}

Aborting the fetch closes the underlying connection, which on a well-behaved async server raises a disconnect that your generator can catch. In FastAPI, checking await request.is_disconnected() periodically inside your loop (or relying on asyncio.CancelledError propagating when the client goes away) is what actually stops the upstream LLM call instead of leaving it running orphaned on the server. Skipping this is how teams end up with a background job quietly generating full answers nobody will ever see, burning provider spend on every abandoned message.

Polishing the perceived experience

Once the mechanics work, most of the remaining effort is UX polish that has an outsized effect on how "finished" the product feels, even though none of it touches the RAG pipeline itself:

A blinking cursor at the end of the answer while tokens are still arriving, removed on the done event, signals "still writing" far more clearly than a static text block that might just be short.
Markdown rendering that updates incrementally rather than waiting for the full answer — parse and render on every token batch so code blocks and lists build up live instead of popping in all at once at the end.
Disabling the input field during generation and re-enabling it (or swapping it for the stop button) prevents users from firing a second query into an already-busy stream.
Auto-scrolling the answer container as tokens arrive, but only if the user hasn't manually scrolled up to re-read an earlier part of the conversation — nothing is more annoying than a chat window that yanks you back down while you're trying to read something above.
Debouncing the source citations panel so it doesn't flicker in in the middle of the answer if your architecture happens to resolve sources slightly before the final token lands.

None of these are hard individually, but skipping them is exactly what makes a technically correct streaming implementation still feel unfinished in a demo.

Handling errors mid-stream

Streams fail differently than regular requests. A normal API call either succeeds or throws before your code moves on. A stream can succeed for the first ten tokens and then die — the model provider times out, your retrieval database drops a connection, a rate limit kicks in mid-generation. If you don't plan for this, the user is left staring at half an answer with no indication anything went wrong.

Wrap the generator in a try/except that emits a distinct error event type, and make sure the frontend has a branch for it:

async def rag_stream(query: str):
    try:
        yield {"type": "status", "message": "Searching documents..."}
        chunks = await retrieve(query)
        context = build_context(chunks)

        async for token in generate_answer(query, context):
            yield {"type": "token", "content": token}

        yield {"type": "done", "sources": [c["source"] for c in chunks]}
    except Exception as e:
        yield {"type": "error", "message": "Something went wrong generating a response."}

Never leak raw exception text to the client in that error message — log the real exception server-side and send a generic, user-facing string over the wire. It's a small habit but it prevents stack traces and internal service names from ending up in a browser console that a curious user might screenshot and post somewhere.

Testing streaming behavior locally

Streaming bugs are notoriously hard to catch because your local machine has near-zero network latency, which hides buffering issues that only appear behind a real reverse proxy or CDN. A few things worth doing before you ship:

Test behind an actual Nginx or CDN config in staging, not just localhost, since buffering behavior differs completely.
Artificially throttle your local network (Chrome DevTools has a network throttling panel) to see how your UI handles slow, chunky token arrival.
Kill the connection mid-stream on purpose — close the tab, disconnect Wi-Fi — and confirm your backend generator actually stops calling the LLM provider instead of continuing to burn tokens for a client that's gone. asyncio.CancelledError handling on the server side matters here.
Send a query that returns zero relevant chunks and verify the model is honest about not knowing, rather than hallucinating from parametric memory.

That third point is one people skip constantly and then get an unpleasant billing surprise: if a user closes the tab mid-answer, your server-side generator needs to detect the disconnect and stop consuming the upstream LLM stream, otherwise you keep paying for tokens nobody's reading.

Closing thoughts

None of the individual pieces here — SSE, async generators, a rerank step, a buffered stream parser — are exotic. What makes RAG streaming responses feel hard is that the failure modes are quiet: a missing proxy header, a status event that never clears, a citation marker that doesn't resolve, a stream that keeps generating after the user has left. Each of these is a small bug, but stacked together they're the difference between a chatbot that feels responsive and one that feels broken in ways you can't quite explain in a bug report. Build the retrieval and generation phases as one typed event stream from the start, treat citation resolution as a post-stream step, and test under real network conditions before you trust a demo. If you're earlier in the RAG journey and want the foundational concepts behind why retrieval works the way it does, our Introduction to RAG course covers the retrieval and indexing fundamentals this article assumes you already have in place.