Streaming Agent Responses to Users: A Practical Implementation Guide

Agent streaming is the practice of pushing an AI agent's output to the user incrementally, token by token and step by step, instead of waiting for the entire run to finish before showing anything. For a simple chat completion this means streaming text as it's generated. For an agent, it means something more: streaming tokens, tool call announcements, tool results, and intermediate reasoning as the agent works through a multi-step task that might take ten or thirty seconds. This guide covers the protocols, the server-side code, the client-side rendering, and the failure modes that only show up once real users hit your endpoint.

Why Agent Streaming Matters More Than Chat Streaming

A single LLM call that streams text is a solved problem: open a connection, forward tokens as they arrive, done. Agent streaming is harder because an agent isn't one call, it's a loop. A typical agent turn looks like: call the model, get back either a text response or a request to call a tool, execute the tool, feed the result back to the model, repeat until the model produces a final answer. Each of those loop iterations can take a few seconds. Without streaming, a user watching a spinner for 15 seconds while the agent searches a database, calls an API, and reasons about the result has no idea whether the system is working or hung.

Agent streaming fixes this by exposing the loop itself, not just the final tokens. A well-instrumented stream tells the client: "calling tool search_orders", then "tool returned 3 results", then starts streaming the model's text response that references those results. This is the difference between a black box and a system the user can trust. It also matters for cost and latency perception: research on interface responsiveness consistently shows that showing partial progress reduces perceived wait time even when total completion time is unchanged. For an agent that might genuinely take 20-40 seconds to finish a task, that perception gap is the difference between a product that feels broken and one that feels fast.

Streaming Protocols: SSE, WebSockets, and Chunked HTTP

Three transport options come up repeatedly when you're building agent streaming into a web product.

Server-Sent Events (SSE) is the default choice for most agent products. It's a one-way stream from server to client over a normal HTTP connection, it works through most corporate proxies and load balancers without special configuration, and every browser has a native EventSource client (though most agent UIs hand-roll a fetch reader instead, for more control over headers and reconnection). SSE is the right choice whenever the client only needs to receive events and all user input (a follow-up message, a cancel action) happens as separate HTTP requests.

WebSockets give you a full duplex connection, useful if the user needs to interrupt or steer the agent mid-run, or if you're streaming from a long-lived session that mixes multiple concurrent agent tasks. The cost is more infrastructure: you need a stateful connection manager, and typical serverless deployments (Lambda, Vercel functions, Cloud Run with request-based scaling) don't hold WebSocket connections well without extra plumbing.

Plain HTTP chunked transfer encoding is what you get if you skip both and just write to the response stream as data becomes available, letting the HTTP/1.1 Transfer-Encoding: chunked mechanism do the work. It's the lowest-level option and it's what SSE is built on top of, so unless you have a strong reason to hand-roll it, use SSE's framing (data: ...\n\n) even over a raw chunked response, because client libraries expect that format.

For most teams building an agent product, start with SSE. Move to WebSockets only when you have a concrete requirement for bidirectional mid-stream communication, such as a voice agent or a coding agent the user wants to interrupt.

Building an Agent Streaming Endpoint with Server-Sent Events

Here's a minimal but production-shaped SSE endpoint using Python and FastAPI. It streams three kinds of events: token deltas, tool call announcements, and a final done event. The pattern generalizes to any agent framework, the important part is the event envelope, not the specific LLM SDK.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import json
import asyncio

app = FastAPI()


async def run_agent_stream(user_message: str):
    # yield a structured event as an SSE frame
    def event(event_type, data):
        payload = json.dumps({"type": event_type, "data": data})
        return f"data: {payload}\n\n"

    yield event("start", {"message": "agent run started"})

    # first model call decides whether to call a tool
    yield event("tool_call", {"name": "search_orders", "args": {"query": user_message}})
    await asyncio.sleep(0.5)  # simulate tool latency
    tool_result = {"orders": [{"id": "A1", "status": "shipped"}]}
    yield event("tool_result", {"name": "search_orders", "result": tool_result})

    # stream the model's final answer token by token
    final_text = "Your most recent order A1 has shipped."
    for word in final_text.split(" "):
        yield event("token", {"text": word + " "})
        await asyncio.sleep(0.05)

    yield event("done", {"finish_reason": "stop"})


@app.post("/agent/stream")
async def agent_stream(payload: dict):
    return StreamingResponse(
        run_agent_stream(payload["message"]),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no",
        },
    )

Two headers matter here and get skipped constantly. Cache-Control: no-cache stops intermediate caches from buffering the whole response before forwarding it. X-Accel-Buffering: no disables response buffering in nginx specifically, which is a common silent killer of streaming: without it, nginx waits for the full response body before sending anything to the client, and your beautifully streamed SSE arrives as one lump at the end. If you're behind nginx or a similar reverse proxy, check this setting before you assume your client-side code is broken.

If you're calling a real LLM API rather than simulating one, the loop is the same shape: call the model with streaming enabled, forward text deltas as token events, and when the model's response includes a tool-use request, pause the text stream, emit a tool_call event, execute the tool, emit tool_result, then continue the loop with the tool result appended to the conversation and call the model again.

Streaming Tool Calls and Intermediate Reasoning Steps

The event types in the example above (tool_call, tool_result, token) are a minimal vocabulary. In practice you'll want a few more for a good agent UX:

• thinking or reasoning, for models and frameworks that expose an intermediate reasoning trace, so the UI can show a collapsed "thinking" indicator instead of blank space
• tool_call_partial, if you want to show tool arguments as they're generated rather than only after the full call is assembled (useful for long arguments like generated SQL or code)
• error, a first-class event type so the client can render a failure inline instead of just dropping the connection
• usage, sent once at the end with token counts, useful for client-side cost display or internal logging without a separate round trip

Keep the event schema flat and versioned from day one. Something like {"type": "token", "data": {...}, "v": 1} costs nothing now and saves you from a painful client/server desync later when you add new event types. Clients should ignore event types they don't recognize rather than erroring, which lets you roll out new event kinds without a hard client update.

Handling Partial JSON During Structured Output Streaming

A common failure case: your agent is streaming a structured tool call argument (say, a JSON object it's building token by token) and the client tries to JSON.parse the partial string on every chunk to show a live preview. This throws constantly, because {"query": "sh isn't valid JSON until the closing brace arrives.

Two practical approaches. The simple one: don't try to parse partial JSON at all, buffer the raw string client-side and only parse once you receive an explicit tool_call_complete event carrying the full argument string. This is the right default for most products, since users rarely need to see a tool's raw arguments mid-generation.

The more ambitious one, needed for coding agents or anything showing live-generated JSON/code to the user: use a streaming-tolerant JSON parser that can handle incomplete input, such as a partial-JSON parsing library, and re-render the best-effort parse on every chunk. This is what gives you the "watch the code appear" effect you see in agent coding tools. It adds real complexity (handling malformed intermediate states, arrays that aren't closed yet, string values that are mid-escape-sequence) so only take this on if the live preview is a genuine product requirement.

Consuming the Stream on the Client

Here's a client-side reader for the SSE endpoint above, written as a plain fetch call rather than EventSource, because EventSource doesn't support custom headers (needed for auth) or POST bodies (needed to send the user's message).

async function streamAgent(message, onEvent) {
  const response = await fetch("/agent/stream", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": `Bearer ${getAuthToken()}`,
    },
    body: JSON.stringify({ message }),
  });

  if (!response.ok || !response.body) {
    throw new Error(`Stream request failed: ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });
    const frames = buffer.split("\n\n");
    buffer = frames.pop(); // keep the last, possibly incomplete frame

    for (const frame of frames) {
      const line = frame.trim();
      if (!line.startsWith("data:")) continue;
      const jsonStr = line.slice(5).trim();
      const evt = JSON.parse(jsonStr);
      onEvent(evt);
    }
  }
}

In a React component, wire this into state that accumulates tokens and shows tool calls as inline status lines:

function useAgentStream() {
  const [messages, setMessages] = React.useState([]);
  const [status, setStatus] = React.useState("idle");

  const send = React.useCallback(async (text) => {
    setStatus("streaming");
    let draft = "";

    await streamAgent(text, (evt) => {
      if (evt.type === "tool_call") {
        setMessages((m) => [...m, { role: "system", text: `Calling ${evt.data.name}...` }]);
      } else if (evt.type === "token") {
        draft += evt.data.text;
        setMessages((m) => {
          const rest = m.filter((msg) => msg.role !== "assistant-draft");
          return [...rest, { role: "assistant-draft", text: draft }];
        });
      } else if (evt.type === "done") {
        setMessages((m) => {
          const rest = m.filter((msg) => msg.role !== "assistant-draft");
          return [...rest, { role: "assistant", text: draft }];
        });
        setStatus("idle");
      }
    });
  }, []);

  return { messages, status, send };
}

The key detail in the frame-parsing loop is the buffer.split("\n\n") / buffer.pop() pattern. TCP and HTTP chunking don't respect your SSE frame boundaries, a single read() call might return half a frame, one and a half frames, or five frames at once. Always buffer and split on the frame delimiter rather than assuming one read() equals one event, this is the single most common bug in hand-rolled SSE clients.

Error Handling and Reconnection

Agent runs fail mid-stream more often than a simple chat completion does, because there's more surface area: a tool call can time out, an upstream API can rate-limit you, the model call itself can fail after tokens have already started flowing. Design for this from the start rather than bolting it on later.

On the server, wrap the agent loop in a try/except that, on failure, emits an explicit error event with a short, user-safe message before closing the stream, rather than just dropping the connection:

async def run_agent_stream(user_message: str):
    try:
        # ... agent loop ...
        yield event("done", {"finish_reason": "stop"})
    except ToolTimeoutError:
        yield event("error", {"message": "A tool call timed out, please try again."})
    except Exception:
        yield event("error", {"message": "Something went wrong processing your request."})

On the client, don't treat a dropped connection as silent success. If the stream ends without a done or error event, that's a network-level failure and the UI should say so rather than leaving a half-written response on screen forever. For reconnection, SSE technically supports the Last-Event-ID header for automatic resume, but agent streams rarely benefit from mid-run resume, since the server-side agent state (the tool call in flight, the partial model generation) usually can't be picked back up from an arbitrary byte offset. The more practical pattern is: on disconnect, show the partial output with a "reconnecting" or "retry" affordance, and let a retry start a fresh agent turn rather than trying to resume the old one.

Backpressure and Rate Limiting Under Streaming Load

Streaming responses hold a connection open for the full duration of the agent run, which is longer than a typical request-response cycle and changes your capacity planning. A server that handles 500 concurrent 200ms API requests comfortably might struggle with 500 concurrent 20-second agent streams, because each one now occupies a worker, a database connection, or an event loop slot for much longer.

A few concrete mitigations. First, put a hard ceiling on concurrent streams per user and return a clear 429 with a retry hint if that ceiling is hit, rather than letting requests queue silently. Second, if you're running behind a process-based server (Gunicorn workers, for example) rather than a fully async one, streaming endpoints need an async-capable worker class, a sync worker will block on the whole stream duration and tank your throughput. Third, set a maximum stream duration server-side (say, 60-120 seconds) and emit a graceful error event if the agent loop runs past it, an agent stuck in a tool-call retry loop should not be able to hold a connection open indefinitely.

Testing and Observability for Streaming Agents

Streaming endpoints are awkward to test with typical request/response assertions, since the interesting behavior is the sequence and timing of events, not just the final payload. A few patterns that hold up:

• Write an integration test that opens the stream, collects every event into a list, and asserts on the sequence of event types (start, tool_call, tool_result, token*, done) rather than just the concatenated final text. This catches regressions where, say, a tool_result event silently stops being emitted.
• Log every event server-side with a shared run ID, so a support ticket about "the agent hung" can be traced to exactly which step it stalled on. Without this, streaming failures are close to undebuggable in production, because the client only sees a spinner and the server logs (if you didn't structure them) show one giant request.
• Track time-to-first-token and time-to-first-tool-call as separate metrics from total run duration. These are the numbers that drive perceived latency, and they can regress independently of total time, for example if you add an expensive pre-processing step before the first model call.
• Test the buffering edge case explicitly: mock a fetch response where the body arrives in deliberately awkward chunk boundaries (splitting a single SSE frame across three read() calls) and confirm your client parser still reconstructs events correctly.

FAQ

Does agent streaming require a different LLM API call than a normal chat completion? No. Most LLM APIs support a streaming flag on the same completion endpoint you'd use for a non-streaming call. What's different for an agent is that you're managing a loop of these calls, some interleaved with tool execution, and you need to define your own event envelope to represent tool calls and results alongside the model's token stream, since the raw model stream only covers the "generate text" half of the picture.

Can I stream an agent's response over a serverless function? Yes, but check your platform's specific limits first. Many serverless runtimes cap execution duration and some historically buffered the full response before returning it, which defeats streaming. Confirm your platform supports streaming responses (not just streaming requests) and test with a deliberately slow endpoint before shipping, rather than assuming it works because the framework docs mention StreamingResponse support.

Should I stream the agent's internal reasoning to the user? It depends on the product. Showing a lightweight "thinking" or "searching" indicator improves perceived responsiveness without exposing raw chain-of-thought. Whether to stream a full reasoning trace verbatim is a product and trust decision as much as a technical one, some users want the transparency, others find a wall of intermediate reasoning noisy. A safe default is to stream compact status events (what tool is running, why) rather than raw model reasoning text.

How do I stream to multiple clients watching the same agent run? Decouple the agent's execution from the stream delivery. Run the agent loop once, publish its events to a lightweight pub/sub channel (an in-memory event bus for a single server, Redis pub/sub or a similar broker across multiple servers), and have each connected client's SSE handler subscribe to that channel and forward events. Don't re-run the agent per connected client, that duplicates cost and can produce diverging results if any step is non-deterministic.

What's the simplest way to add streaming to an agent that doesn't have it today? Start by streaming just the final text response with a normal SSE endpoint, without touching tool call visibility. That alone removes the "long silent wait" problem for the common case where most of an agent's time is spent generating the final answer. Once that's shipped and stable, add tool_call and tool_result events as a second iteration, since those require more careful event-schema design and testing.