Streaming LLM Responses with SSE in Production

LLM streaming SSE is the standard way to push model tokens to a browser the moment they are generated, over a single long-lived HTTP response using the Server-Sent Events format. You use it because waiting eight seconds for a full completion feels broken, while first tokens in 300ms feels instant. This guide covers the wire format, a runnable server and client, and the production traps: proxy buffering, cancellation, backpressure, and reconnection, that never show up in a local demo.

Why LLM streaming SSE instead of polling or WebSockets

A chat completion is generated one token at a time. If you wait for the whole thing and return a single JSON body, the user stares at a spinner for the full generation time. Streaming flips this: you flush each token as the model produces it, so perceived latency drops to time-to-first-token instead of time-to-last-token. For a 600-token answer that is the difference between one long pause and a response that types itself out.

There are three transports people reach for. Polling means the client asks "done yet?" on a timer, which is wasteful and always a beat behind. WebSockets give you a full duplex socket, which is more than you need: model output flows one direction, server to client, and you pay for connection upgrade handling, heartbeats, and a separate protocol. Server-Sent Events sit in the middle. It is plain HTTP, one direction, text frames, with automatic reconnection built into the browser. That is exactly the shape of an LLM response, which is why every major provider streams over an SSE-style text/event-stream body.

The one real constraint of SSE is that it is unidirectional and text only. You cannot send binary frames, and the client cannot talk back on the same channel. For token streaming that is fine. The client sends one POST to start generation and reads the stream in the response; anything it needs to say mid-stream (like "stop") goes over a separate request or by aborting the connection.

The SSE wire format, exactly

SSE is a line-based text protocol. The whole spec that matters for streaming fits in a few rules. A message is one or more lines, and a blank line dispatches it. Each line is a field: value pair. The fields you actually use are data, event, and id.

A minimal event looks like this:

data: Hello

data: world

Two events, each a single data line, each terminated by a blank line. The browser's EventSource fires a message event for each, with event.data set to Hello then world.

Multi-line data concatenates with newlines. This matters because your JSON payloads must not contain raw newlines mid-value, or you will split one logical message across data fields by accident:

data: {"token": "hel"}
data: {"token": "lo"}

That is one event whose data is the two lines joined by \n, which is almost never what you want. Keep each JSON object on a single data: line.

You can name events with the event field and attach a resume token with id:

event: token
id: 42
data: {"text": "def "}

event: done
data: {"finish_reason": "stop"}

Comments are lines that start with a colon. They are ignored by the client but keep the connection warm and defeat some proxy idle timeouts:

: keep-alive

Two conventions are worth adopting. First, send an explicit end marker rather than relying on the connection closing. The OpenAI-compatible convention is a final data: [DONE] line. Second, decide up front whether you send raw text or JSON in data. JSON is more work but lets you carry token text, usage counts, tool-call deltas, and finish reasons in one channel. In production, use JSON.

A runnable server: Node with the Anthropic SDK

Here is a complete streaming endpoint using plain Node http and the official @anthropic-ai/sdk. It streams Claude's output to the browser as SSE. The provider SDK gives you an async iterator of events; your job is to reframe each text delta as an SSE frame and flush it.

import http from "node:http";
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic(); // reads ANTHROPIC_API_KEY

http.createServer(async (req, res) => {
  if (req.method !== "POST" || req.url !== "/chat") {
    res.writeHead(404).end();
    return;
  }

  const body = await readJson(req);

  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache, no-transform",
    "Connection": "keep-alive",
    "X-Accel-Buffering": "no",
  });

  const send = (event, data) => {
    res.write(`event: ${event}\n`);
    res.write(`data: ${JSON.stringify(data)}\n\n`);
  };

  const abort = new AbortController();
  req.on("close", () => abort.abort());

  try {
    const stream = client.messages.stream(
      {
        model: "claude-sonnet-4-5",
        max_tokens: 1024,
        messages: body.messages,
      },
      { signal: abort.signal }
    );

    for await (const chunk of stream) {
      if (chunk.type === "content_block_delta" &&
          chunk.delta.type === "text_delta") {
        send("token", { text: chunk.delta.text });
      }
    }
    send("done", { finish_reason: "stop" });
  } catch (err) {
    if (!abort.signal.aborted) {
      send("error", { message: String(err?.message || err) });
    }
  } finally {
    res.end();
  }
}).listen(3000);

function readJson(req) {
  return new Promise((resolve, reject) => {
    let raw = "";
    req.on("data", (c) => (raw += c));
    req.on("end", () => {
      try { resolve(JSON.parse(raw || "{}")); }
      catch (e) { reject(e); }
    });
  });
}

Four things in the header block are load-bearing. Content-Type: text/event-stream is what makes it SSE. Cache-Control: no-transform tells intermediaries not to rewrite the body. X-Accel-Buffering: no disables buffering in nginx specifically, which is the single most common reason streaming works locally and dies behind a reverse proxy. And you must not set Content-Length, because the body length is unknown; Node handles this with chunked transfer encoding automatically as long as you never set it.

The AbortController wired to req.on("close") is the cancellation path. When the browser tab closes or the user hits stop and the fetch is aborted, the socket closes, req emits close, and you abort the upstream provider request. Skip this and you keep paying for tokens the user will never see, on every abandoned generation, which at scale is real money.

The browser client: fetch plus a stream reader

The classic client is the built-in EventSource, but it only does GET and cannot set an Authorization header or send a request body. Real chat needs POST with a JSON body and auth, so most production clients parse SSE by hand over fetch and a ReadableStream reader. It is not much code.

async function streamChat(messages, onToken) {
  const res = await fetch("/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages }),
  });

  const reader = res.body
    .pipeThrough(new TextDecoderStream())
    .getReader();

  let buffer = "";
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += value;

    // Events are separated by a blank line.
    const events = buffer.split("\n\n");
    buffer = events.pop(); // keep the incomplete trailing chunk

    for (const raw of events) {
      const lines = raw.split("\n");
      let event = "message";
      let data = "";
      for (const line of lines) {
        if (line.startsWith("event:")) event = line.slice(6).trim();
        else if (line.startsWith("data:")) data += line.slice(5).trim();
      }
      if (!data) continue;
      if (event === "token") onToken(JSON.parse(data).text);
      if (event === "done") return;
    }
  }
}

The critical bug this code avoids: network chunks do not align with SSE events. A single reader.read() might hand you two and a half events, or half of one. The buffer accumulates bytes, you split on the \n\n delimiter, and you deliberately keep the last (possibly incomplete) fragment in the buffer for the next read. Beginners who call JSON.parse directly on each chunk get intermittent parse errors under load and cannot reproduce them locally, because on localhost the whole response often arrives in one chunk. The bug only appears when the network fragments the stream.

To let the user cancel, thread an AbortController into the fetch and call abort() on a stop button. That closes the socket, which fires close on the server, which aborts the provider call. The whole cancellation chain is that one signal propagated end to end.

Production trap 1: proxy and CDN buffering

This is the failure that eats the most hours. Your stream works perfectly against localhost, you deploy behind nginx or a CDN or a serverless platform, and suddenly the whole response arrives at once after the model finishes. The tokens were generated incrementally; something in the middle buffered them into one lump.

nginx buffers proxied responses by default. Fix it on the location block with proxy_buffering off and proxy_cache off, and make sure proxy_read_timeout is longer than your worst-case generation. The X-Accel-Buffering: no response header does the same job per response, which is handy when you cannot edit the nginx config. Apache with mod_proxy needs similar flush handling.

CDNs are worse because many will not stream text/event-stream at all on certain tiers, or they buffer to apply compression. Two rules: do not gzip an SSE stream (compression buffers to fill its window, which defeats the point), and confirm your CDN passes text/event-stream through untouched. Cloudflare streams SSE but you may need to disable certain optimization features on the route.

Serverless adds its own wrinkle. Traditional request-response function models buffer the whole body before returning, so naive SSE on them does not stream at all. Platforms now offer streaming-capable runtimes (edge runtimes, or explicit streaming responses on the newer function models). If you are on serverless, verify your platform supports response streaming before you build on it, because no amount of correct header-setting fixes a runtime that buffers by design.

How to test it honestly: curl -N https://yourapp/chat and watch. -N disables curl's own buffering. If tokens dribble out, you are streaming. If the response lands all at once after a pause, something upstream is buffering and no browser change will fix it.

Production trap 2: backpressure and slow clients

A fast model plus a slow client is a memory leak waiting to happen. res.write() in Node returns false when the internal buffer is full, meaning the client is not draining fast enough. If you ignore that return value and keep writing, Node queues the unsent data in memory. A mobile client on a bad connection with a chatty model can make a single request balloon your process memory.

The correct pattern is to respect the return value and wait for the drain event before writing more:

function write(res, chunk) {
  if (!res.write(chunk)) {
    return new Promise((resolve) => res.once("drain", resolve));
  }
  return Promise.resolve();
}

Await that in your loop. Now a slow client naturally slows your read from the provider (or fills a bounded buffer you can cap), instead of accumulating unbounded memory. This almost never shows up in testing because your test client drains instantly on the same machine. It shows up in production as slow memory growth under real traffic, and it is miserable to diagnose after the fact. Wire it in from the start.

Production trap 3: cancellation and cost control

Every generation a user abandons is money spent for nothing if you keep the upstream call alive. The full chain must be connected: browser abort -> fetch socket closes -> server req emits close -> AbortController.abort() -> provider SDK cancels the HTTP call to the model. Break any link and you leak generations.

Test this deliberately. Start a long generation, close the tab, and confirm in your logs that the upstream request was aborted, not that it ran to completion in the background. Also handle the inverse: the provider errors mid-stream. You have already sent a 200 and some tokens, so you cannot change the status code. Send an error event in the stream and let the client render a graceful "generation interrupted" state. Do not just close the socket silently, because the client cannot tell a clean finish from a mid-stream crash without an explicit done or error marker. That end marker is not optional politeness; it is how the client knows the difference.

Production trap 4: reconnection, timeouts, and heartbeats

Long streams die. Load balancers and proxies cut idle connections, mobile networks drop, laptops sleep. Two defenses.

A heartbeat keeps the connection from looking idle. Send an SSE comment line every 15 to 30 seconds during any gap in real tokens:

res.write(": ping\n\n");

It costs nothing, the client ignores it, and it resets idle timers on every hop. Set your proxy_read_timeout (or equivalent) comfortably above the heartbeat interval.

For genuine reconnection, SSE has the id field and the Last-Event-ID header. The browser's native EventSource automatically resends the last seen id on reconnect, and you can resume from there. But token streams are hard to resume meaningfully: you cannot re-run half a generation, and the model is not deterministic across calls. In practice most teams treat a dropped LLM stream as a failed request and let the user retry the whole message rather than trying to splice a partial completion. Reserve true resumption for cases where you persist the accumulated text server-side and can replay what was already generated. Know which mode you are in before you promise reconnection in the UI.

A production checklist

Before you ship LLM streaming SSE, walk this list:

1. Response headers set text/event-stream, no-cache, no-transform, and X-Accel-Buffering: no. Content-Length is never set.
2. curl -N against the deployed URL shows tokens arriving incrementally, not in one lump.
3. The client buffers partial chunks and splits on \n\n before parsing, never JSON.parse on a raw read.
4. An explicit done and error event marks stream end; the client distinguishes clean finish from crash.
5. Client abort propagates all the way to a provider-side AbortController, verified in logs.
6. res.write backpressure is respected with a drain await or a bounded buffer.
7. A heartbeat comment fires every 15 to 30 seconds, and proxy read timeouts exceed it.
8. Compression is off for the SSE route, and your CDN or serverless runtime is confirmed to support streaming.

Get those eight right and streaming behaves the same in production as it does on your laptop, which is the entire point.

FAQ

Should I use SSE or WebSockets for LLM streaming? Use SSE for token streaming. Model output flows one direction, server to client, which is exactly what SSE is built for, and it rides on plain HTTP with automatic browser reconnection. Reach for WebSockets only when you need true bidirectional low-latency messaging on the same channel, like collaborative editing or live audio, where the client streams data back continuously. For chat, the client sends one request and reads the stream; that does not need a duplex socket.

Why does streaming work locally but not in production? Almost always buffering by something between your server and the browser. nginx buffers proxied responses by default, CDNs may buffer to compress, and traditional serverless function models buffer the whole body before returning. Locally there is nothing in the middle, so it works. Fix it with proxy_buffering off, the X-Accel-Buffering: no header, disabling compression on the route, and confirming your hosting runtime supports response streaming. Verify with curl -N.

Can I use the browser's built-in EventSource? Only if a GET request with no custom headers and no request body works for you. EventSource cannot POST, cannot set an Authorization header, and cannot send a JSON body, which most authenticated chat endpoints require. The common production pattern is to parse SSE by hand over fetch and a ReadableStream reader, which is a few dozen lines and gives you full control over method, headers, body, and cancellation.

How do I stop a generation and stop paying for it? Wire an AbortController into the client fetch and abort it on your stop button. Aborting closes the socket, which fires a close event on the server request, which you use to abort a server-side AbortController passed into the provider SDK call. That cancels the upstream request to the model. Test it by starting a long generation, closing the tab, and confirming in logs that the provider call was actually cancelled rather than running to completion in the background.

What is backpressure and why does it matter for streaming? Backpressure is the signal that a client cannot receive data as fast as you can send it. In Node, res.write() returns false when its buffer is full. If you ignore that and keep writing, unsent tokens queue in your process memory, and a slow client with a fast model can grow that queue without bound. Respect the return value: wait for the drain event before writing more, so a slow reader slows your upstream read instead of leaking memory. It rarely shows in testing because local clients drain instantly.

Do I need to send a special end-of-stream marker? Yes. Send an explicit done event (or the OpenAI-style data: [DONE] line) when generation finishes cleanly, and an error event if it fails mid-stream. Without an explicit marker, the client cannot tell a normal completion from a dropped connection or a server crash, because in both cases the socket simply closes. The marker is what lets the UI show "done" versus "interrupted, retry" correctly.