Batching and Streaming: Two Ways to Serve LLM Requests

The Moment You Realize Serving Is Harder Than Prompting

You wrote a clever prompt, wired it to an API, and the demo felt like magic. Then real users showed up. Some of them wanted answers to appear the instant they hit enter. Others were running overnight jobs that needed to process a hundred thousand documents without anyone watching. Suddenly the same model, the same prompt, and the same code had two completely different jobs to do, and the way you fed requests into the model started to matter more than the prompt itself.

This is where two words enter your vocabulary and never leave: batching and streaming. They sound like opposites, and in some ways they are, but they answer two different questions. Batching asks how you pack many requests together so the hardware stays busy and the cost per request drops. Streaming asks how you deliver a single response so the human on the other end feels like something is happening. Get these two ideas right and your system feels fast, cheap, and reliable. Get them wrong and you either burn money or make users stare at a spinner. This article walks through both, when to reach for each, and how they interact in a real production stack.

Batching: Feeding The Model In Bulk

Batching means grouping multiple requests and processing them together in a single pass through the model. The reason this matters comes down to how modern accelerators like GPUs work. A GPU is a machine built for doing thousands of small math operations at the same time. When you send it a single request, most of those parallel lanes sit idle. The model still has to load its weights from memory, and that loading cost is roughly the same whether you process one request or thirty-two. So if you can fill those idle lanes with more requests, you get far more work done for nearly the same fixed cost.

Think of it like a bus versus a taxi. A taxi picks up one passenger and drives off, which is fast for that passenger but expensive per seat. A bus waits until it fills up, then moves everyone at once. The bus is more efficient per passenger, but the first person to board waits for the others. That trade-off, efficiency versus waiting, is the heart of batching.

There are a few flavors of batching worth naming clearly:

• Static batching collects a fixed number of requests, runs them together, and returns them together. Simple to reason about, but the whole batch moves at the speed of its slowest member.
• Dynamic batching waits a short window, gathers whatever requests arrive in that window, and processes them. It balances waiting time against batch size.
• Continuous batching, sometimes called in-flight batching, is the modern favorite for LLMs. New requests can join the batch as older ones finish, so the accelerator never sits idle waiting for the slowest request to complete.

Here is a stripped-down illustration of dynamic batching on the server side, where you wait a small window to accumulate requests before firing them at the model:

import asyncio

class DynamicBatcher:
    def __init__(self, model, max_batch=16, window_ms=10):
        self.model = model
        self.max_batch = max_batch
        self.window = window_ms / 1000
        self.queue = asyncio.Queue()

    async def submit(self, prompt):
        future = asyncio.get_event_loop().create_future()
        await self.queue.put((prompt, future))
        return await future

    async def run(self):
        while True:
            prompt, fut = await self.queue.get()
            batch = [(prompt, fut)]
            # Collect more requests until window expires or batch is full
            try:
                deadline = asyncio.get_event_loop().time() + self.window
                while len(batch) < self.max_batch:
                    timeout = deadline - asyncio.get_event_loop().time()
                    if timeout <= 0:
                        break
                    item = await asyncio.wait_for(self.queue.get(), timeout)
                    batch.append(item)
            except asyncio.TimeoutError:
                pass

            prompts = [p for p, _ in batch]
            results = self.model.generate(prompts)  # one bulk call
            for (_, future), result in zip(batch, results):
                future.set_result(result)

The key idea in that snippet is the window. You accept a tiny bit of extra latency, ten milliseconds here, in exchange for the chance to bundle several requests into one model call. On a busy server that window fills instantly and you get large batches. On a quiet server it expires quickly and single requests still go through with minimal delay.

Streaming: Delivering Tokens As They Arrive

Streaming is a different animal. A language model generates its answer one token at a time, left to right. Without streaming, your server waits for the entire answer to finish, then sends the whole block back at once. The user watches nothing happen for several seconds, then the full paragraph appears. With streaming, you send each token to the client the moment the model produces it, so words appear on screen the way a person types them.

The metric that matters most for streaming is time to first token, often shortened to TTFT. This is how long the user waits before the very first word shows up. A response that takes eight seconds to fully generate can still feel snappy if the first token lands in three hundred milliseconds, because the user immediately sees progress and starts reading. Human perception of speed is not about total time, it is about how quickly something starts and whether it keeps moving.

Under the hood, streaming almost always rides on top of server-sent events, a simple protocol where the server keeps a connection open and pushes small chunks of data as they become ready. Here is what consuming a streamed completion looks like on the client, using the pattern most modern LLM APIs follow:

from openai import OpenAI

client = OpenAI()

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain batching in one paragraph."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)

Notice that there is no single moment where the whole answer arrives. Instead you loop over chunks, each carrying a small delta of text, and you print it right away. The flush=True matters because it forces the output to appear immediately instead of sitting in a buffer. On a web frontend, the same loop would append each chunk to a growing message bubble, which is exactly how a chat interface produces that familiar typing effect.

Streaming does not make the model faster. The total generation time is the same. What changes is the human experience of that time, and in interactive products that experience is often the whole ballgame.

They Are Not Opposites, They Are Layers

Here is the insight that trips up a lot of engineers early on. Batching and streaming sound mutually exclusive, as if you must pick one. In reality they operate at different layers of the system and combine happily. Batching is about how the server schedules work internally. Streaming is about how the server delivers results externally. You can absolutely run a server that batches many requests together for efficiency while streaming each of those requests back to its own user token by token.

Continuous batching makes this especially natural. Imagine four users each send a chat message at nearly the same time. The server places all four into a single running batch on the GPU. On every generation step, the model produces one new token for each of the four sequences at once. The server then fans those tokens out, sending user one their token, user two their token, and so on. Each user sees a smooth stream, while the hardware enjoys the efficiency of processing four requests in parallel. Nobody waits for a bus that is idling, and nobody rides an expensive taxi alone.

So the real mental model is a stack:

1. Delivery layer (streaming) decides how a finished-or-in-progress response reaches the client.
2. Scheduling layer (batching) decides how many requests share a single model pass.
3. Model layer does the actual token generation, indifferent to both concerns above.

Once you see it as layers rather than a binary choice, most of the confusion dissolves. The question is never batching or streaming. The question is which batching strategy and whether to stream, decided independently.

Latency, Throughput, And Why You Cannot Have It All

Every serving decision lives on a triangle of latency, throughput, and cost, and you rarely optimize all three at once. It helps to define the terms cleanly before deciding.

• Latency is how long one request takes from send to complete. Lower is better for users.
• Throughput is how many requests the system finishes per second across everyone. Higher is better for your bill.
• Cost is dollars per request, driven largely by how well you keep the hardware busy.

Batching pushes throughput up and cost down by keeping the accelerator saturated, but it can nudge individual latency up because requests sometimes wait to be grouped. Streaming leaves total latency untouched but dramatically improves perceived latency by delivering early. If you chase the absolute lowest latency for a single request, you batch less and waste hardware. If you chase the lowest cost, you batch aggressively and some requests wait a little longer. There is no setting that wins every axis, only a setting that fits your workload.

A blunt but useful way to think about the batching sweet spot:

Too little batching:
  GPU sits idle, cost per request is high, latency is great.

Too much batching:
  GPU is saturated, cost per request is low, but the queue grows
  and tail latency (the slowest 1% of requests) gets painful.

The goal:
  Batch enough to keep the GPU busy, but bound the wait so the
  slowest requests still finish within your latency budget.

The phrase to burn into memory is tail latency. Averages lie. A system can have a wonderful average response time while a small slice of unlucky users waits far too long because they got stuck behind a giant batch. Good serving setups cap batch sizes and wait windows specifically to protect that tail, accepting slightly lower peak throughput in exchange for a promise that no request gets abandoned in a queue.

Choosing Based On The Shape Of Your Workload

The right approach falls out naturally once you describe what your traffic actually looks like. Let me sketch the common shapes.

Interactive chat and copilots. A human is watching and waiting. Here streaming is close to mandatory because perceived speed dominates satisfaction, and continuous batching underneath lets you serve many concurrent users affordably. This is the classic pairing: stream out, batch continuously in.

Bulk offline processing. You are classifying a million reviews, generating embeddings for a document store, or summarizing an archive overnight. No human waits on any single result. Throughput is everything and latency per item is irrelevant. Here you batch as aggressively as your memory allows and you almost never stream, because there is no viewer to benefit from early tokens. You just want the whole job done cheaply.

Real-time APIs behind other services. Another program calls your endpoint and needs a structured result, maybe JSON, before it can proceed. Streaming often adds little value because the caller cannot act on half a JSON object, so it waits for completion anyway. Moderate dynamic batching usually wins, balancing latency for the calling service against efficiency.

A rough decision guide:

• Is a human watching the response render? If yes, stream.
• Does the consumer need the full, parsed result before acting? If yes, streaming buys little.
• Is per-request latency part of your promise? If yes, cap batch size and wait windows tightly.
• Is total cost the dominant concern and latency negligible? If yes, batch as large as memory permits.

You can mix these within one product. A support tool might stream the chat reply to the agent while, in a separate offline pipeline, batch-summarizing yesterday's tickets with no streaming at all. Same model, two serving strategies, chosen by the shape of each task rather than by habit.

Practical Pitfalls That Bite In Production

A few hard-won lessons tend to catch teams the first time they run this for real, so it is worth naming them plainly.

Streaming breaks naive error handling. When you commit to a streamed response, you have already sent the user the first half of an answer before you discover the model produced something wrong or the connection dropped mid-flight. You cannot take those tokens back. Build your clients to handle a stream that stops partway, and decide in advance whether you retry, show a graceful truncation notice, or fall back to a fresh non-streamed attempt.

Batching can starve small requests. If a huge prompt and a tiny prompt land in the same batch, the tiny one waits for the huge one to finish generating, because the batch advances together step by step. Continuous batching softens this, but you still want to watch for cases where a long generation holds up short ones. Some teams route very long requests to a separate pool so they cannot block the quick, interactive traffic.

Timeouts interact badly with both. A streamed connection held open for a minute can trip proxy or load-balancer timeouts that were tuned for short requests. And a request waiting in a batch queue is spending part of its timeout budget doing nothing. Make sure your timeout accounting includes queue time, not just generation time, or you will see mysterious failures that never reproduce in a quiet test environment.

Measuring the wrong thing. If you only track average latency, you will feel great while a slice of users suffers. Track time to first token separately from total latency, and track the tail, the ninety-fifth and ninety-ninth percentiles, not just the mean. The numbers that matter for user happiness and the numbers that matter for cost are different numbers, and a healthy dashboard shows both.

Here is a compact mental checklist you can keep near your serving code:

Before shipping an LLM endpoint, confirm you know:
  - TTFT target       (how fast the first token must appear)
  - total latency SLO (how slow the slowest acceptable request is)
  - max batch size    (bounded so the tail stays inside the SLO)
  - queue timeout      (counted as part of the request budget)
  - stream failure path (what the client does on a mid-stream drop)

If you can answer all five for your endpoint, you understand your serving posture. If any answer is a shrug, that is the gap that will page you at 2 a.m.

Bringing It Together

Batching and streaming are not rival techniques you choose between. They are two different levers on two different parts of the same system. Batching lives on the inside, deciding how requests share the expensive hardware so your cost per answer stays sane and your throughput stays high. Streaming lives on the outside, deciding how each answer reaches its reader so the experience feels immediate even when the underlying work takes seconds. The best production systems reach for both at once, batching continuously for efficiency while streaming individually for delight, and they tune the wait windows and batch sizes to protect the slowest unlucky request rather than flattering the average.

The mental model to carry forward is simple. Ask who is waiting and what they need. A human watching a chat wants tokens now, so you stream. A GPU sitting half-idle wants more work, so you batch. A calling service that needs a whole JSON blob does not care about early tokens, so you skip streaming and batch moderately. Once you frame every serving decision as a question about the shape of the workload rather than a fixed rule, the right architecture stops being a guess and starts being obvious.

If you want to go deeper into how real LLM systems are architected, from serving and scaling to evaluation, cost control, and deployment, that is exactly the ground covered in the AI Engineering Roadmap course on teachyou.ai. It walks you through building production-grade AI systems end to end, so the concepts in this article become muscle memory instead of theory. Serving is one of those skills that separates a working demo from a product people trust, and it is well worth the time to master.