Latency vs Throughput in LLM Applications: Why Both Matter

The Two Numbers That Decide Whether Your LLM App Feels Fast

Ship an LLM feature and two very different complaints will land in your inbox. One user says the chat "takes forever to start typing." Another says your batch summarization job "can only handle a handful of documents before it falls over." These are not the same problem, and fixing one can quietly make the other worse. The first is a latency problem. The second is a throughput problem. Most engineers new to LLM systems treat them as a single vague notion of "speed," and that confusion is exactly why their apps feel sluggish, cost too much, or buckle under real traffic.

Latency is about one request. Throughput is about many requests. An LLM serving stack is one of the rare places where these two goals actively fight each other, because the same lever that makes a single response arrive faster often reduces how many responses you can serve per second. If you do not understand the trade-off, you will tune blindly, ship a config that looks fine in a demo, and get paged when a hundred users show up at once. This article breaks down what each metric actually measures in the context of large language models, why they conflict, and how to reason about tuning both at the same time.

What Latency Actually Means for an LLM

Latency is the time between sending a request and getting the result you care about. For a traditional API, that is one number: request in, response out. For an LLM that streams tokens, latency splits into pieces, and you need all of them to describe the experience honestly.

The first piece is time to first token, usually written TTFT. This is how long the user stares at a blank screen or a spinner before the very first word appears. TTFT is dominated by the prefill phase, where the model reads your entire prompt and builds the internal state (the key-value cache) it needs before it can generate anything. A long prompt, a big system message, or a fat retrieval-augmented context all inflate prefill, and therefore inflate TTFT.

The second piece is inter-token latency, sometimes called time per output token. Once generation starts, the model emits tokens one at a time in the decode phase. Inter-token latency is the gap between consecutive tokens. If it is small, text streams smoothly and feels alive. If it is large, words stutter out and the app feels like it is thinking too hard.

Put those together and you get end-to-end latency, the total wall-clock time until the full answer is done. A rough way to think about it:

end_to_end_latency ≈ TTFT + (output_tokens × inter_token_latency)

This little formula already tells you something useful. A response with a short prompt but a long answer is bottlenecked by decode, so inter-token latency matters most. A response with a huge prompt but a one-line answer is bottlenecked by prefill, so TTFT dominates. Averaging everything into one "latency" number hides which phase is actually hurting you.

What Throughput Actually Means for an LLM

Throughput is a rate. It answers the question: how much work can this system finish per unit of time? For LLM serving, people express it two ways, and mixing them up causes real confusion.

The first is requests per second, or how many complete generations the system finishes each second. This is the number your product and capacity planning care about, because it maps directly to "how many users can we serve."

The second is tokens per second, the total number of tokens (across all concurrent requests) the hardware pushes through each second. This is the number that reflects how efficiently you are using the GPU. A single request might decode at 60 tokens per second, but a well-batched server running 32 requests at once might push 3,000 tokens per second in aggregate, even though each individual stream feels no faster.

That gap between per-request speed and aggregate speed is the heart of the whole topic. The GPU does not get faster when you add more concurrent requests. What changes is utilization. LLM decode is memory-bandwidth bound: for each token, the hardware loads billions of model weights out of memory and does relatively little math with them before moving on. When only one request is in flight, most of the GPU's compute sits idle while it waits on memory. When many requests share that same weight load, the fixed cost of reading the weights is amortized across all of them, and total tokens per second climbs dramatically. That mechanism is called batching, and it is the single most important idea in LLM serving performance.

Why the Two Metrics Fight Each Other

Here is where new engineers get burned. The intuitive assumption is that a faster server is faster for everyone, all the time. With LLMs, the opposite is often true: the configuration that maximizes throughput actively worsens latency for the individual request, and vice versa.

Think about batching from a single user's point of view. To build a big batch, the server waits a few milliseconds to collect other incoming requests so they can all run together. That wait is pure added latency for the request that arrived first. Bigger batches mean better GPU utilization and higher tokens per second overall, but every request pays a queueing tax to get into the batch. Push batch size high enough and each request also decodes slightly slower, because the GPU is now genuinely busy serving everyone at once instead of dedicating itself to one stream.

You can feel the trade-off in three regimes:

• Batch size of one. Lowest possible latency for that request. The GPU is almost entirely idle on compute, so tokens per second across the server is terrible and cost per token is high. Great for a latency demo, ruinous for a bill.
• Small to medium batch. Latency rises modestly, throughput rises sharply. This is usually the sweet spot for interactive products, because users barely notice a small latency increase but you serve many times more of them per GPU.
• Very large batch. Throughput approaches the hardware ceiling and cost per token bottoms out, but individual latency degrades enough that interactive users start to complain. This is where offline and batch workloads want to live.

There is a second, harder limit lurking underneath: memory. Every active request holds a key-value cache in GPU memory that grows with its context length. The KV cache, not raw compute, is usually what caps how many requests you can batch. Long prompts and long conversations eat memory fast, so a server that batches 64 short requests comfortably might only fit 8 long-context ones before it runs out of room and has to start queueing. This is why context length quietly governs both your latency and your throughput ceiling at the same time.

A Concrete Way to Measure Both

You cannot tune what you refuse to measure, and averages lie. The most common mistake is reporting mean latency, which hides the tail where your angry users live. Always look at percentiles: p50 tells you the typical experience, p95 and p99 tell you what your least lucky requests feel. A system with a great average and a horrible p99 will generate support tickets all day.

Here is a compact load-test sketch that fires concurrent requests and records the metrics that actually matter. It is deliberately provider-agnostic pseudocode you can adapt to any streaming client:

import asyncio
import time
from statistics import mean, quantiles

async def timed_request(client, prompt):
    start = time.perf_counter()
    first_token_at = None
    output_tokens = 0

    async for chunk in client.stream(prompt):
        if first_token_at is None:
            first_token_at = time.perf_counter()
        output_tokens += 1

    end = time.perf_counter()
    return {
        "ttft": first_token_at - start,
        "end_to_end": end - start,
        "output_tokens": output_tokens,
        "tps_per_request": output_tokens / (end - first_token_at),
    }

async def load_test(client, prompt, concurrency, total):
    sem = asyncio.Semaphore(concurrency)
    results = []

    async def worker():
        async with sem:
            results.append(await timed_request(client, prompt))

    wall_start = time.perf_counter()
    await asyncio.gather(*(worker() for _ in range(total)))
    wall = time.perf_counter() - wall_start

    ttfts = sorted(r["ttft"] for r in results)
    total_tokens = sum(r["output_tokens"] for r in results)

    p95_ttft = quantiles(ttfts, n=100)[94]
    print(f"requests/sec       : {total / wall:.2f}")
    print(f"aggregate tokens/s : {total_tokens / wall:.0f}")
    print(f"mean TTFT (s)      : {mean(ttfts):.3f}")
    print(f"p95 TTFT (s)       : {p95_ttft:.3f}")

Run this at several concurrency levels: 1, 8, 32, 64. Watch what happens. At concurrency 1 your TTFT is beautiful and your requests-per-second is pitiful. As concurrency climbs, aggregate tokens per second rises while p95 TTFT creeps up. The concurrency level where TTFT starts to spike but throughput has mostly flattened is your practical operating point. That crossover, not a number from a vendor's marketing page, is what you tune against.

One more thing this harness will surface if you let it run long enough: the gap between your synthetic prompt and real traffic. A load test that fires the same short prompt every time will report rosy numbers, because every request holds a tiny KV cache and the server batches them effortlessly. Production rarely looks like that. Real users send prompts of wildly different lengths, some with a paragraph of context and some with a thirty-message conversation history, and that variance is where servers fall over. When a handful of long-context requests land at once, they consume the memory budget that dozens of short requests would have shared, batching collapses, a queue forms, and p99 latency detonates while your average still looks fine. So test with a realistic distribution of prompt and output lengths, not a single canned example. The metric that predicts a 2 a.m. page is p99 under mixed load, and it is almost always worse than the number your happy-path benchmark shows.

How to Lower Latency Without Wrecking Throughput

Most latency wins do not require sacrificing throughput at all, because they attack wasted work rather than the batching trade-off. Reach for these first.

• Cut your prompt. Prefill cost scales with input length, so trimming a bloated system prompt or over-stuffed retrieval context directly lowers TTFT. Retrieve five relevant chunks, not fifty. This is the highest-leverage, lowest-effort fix and it also raises throughput because shorter contexts free KV cache memory.
• Use prompt caching. If a large chunk of your prompt is identical across requests (a long system message, a fixed instruction block, few-shot examples), many providers and serving stacks can cache its prefill state and skip recomputing it. TTFT for cache hits drops sharply and you pay less per call.
• Stream the output. Streaming does not change end-to-end latency, but it makes TTFT the number users feel instead of the full completion time. A response that takes eight seconds total but starts streaming in 400 milliseconds feels responsive. The same response delivered all at once at the eight-second mark feels broken.
• Cap output length. Decode dominates end-to-end latency for long answers. Setting a sane maximum and instructing the model to be concise removes tokens you were paying for in both time and money.
• Right-size the model. A smaller or distilled model decodes faster per token and fits more requests per GPU. Route easy requests to a small model and reserve the large one for genuinely hard queries. This helps latency and throughput simultaneously.
• Speculative decoding. A small draft model proposes several tokens that the large model verifies in one pass, which can cut inter-token latency for the big model without changing its output. It costs extra compute, so it helps most when you have spare GPU headroom.

Notice that only the last item touches the batching trade-off. The rest are free wins: less work in, faster and cheaper out.

How to Raise Throughput Without Killing the Experience

When the goal is serving more traffic per dollar, you are working the batching and utilization levers. The good news is that modern serving stacks do most of the heavy lifting if you let them.

• Use continuous batching. Older "static" batching waited for every request in a batch to finish before starting the next batch, so one long generation stalled everything behind it. Continuous batching (also called in-flight batching) adds and evicts requests token by token, so a finished request leaves immediately and a new one takes its slot. This single feature, standard in serving engines like vLLM and TensorRT-LLM, can multiply real-world throughput several times over compared to naive batching. If you take one throughput lesson from this article, take this one.
• Tune the batch and concurrency ceiling deliberately. Set the maximum number of concurrent sequences based on your measured crossover point, not a guess. Too low and the GPU starves; too high and latency and memory blow up.
• Manage the KV cache aggressively. Techniques like paged attention (vLLM's signature feature) stop the cache from fragmenting memory, letting you pack far more concurrent requests into the same GPU. More concurrent requests means higher throughput at the same latency.
• Quantize the model. Running weights at lower precision shrinks the memory the weights occupy and reduces the bandwidth cost of loading them each decode step. That frees memory for a bigger KV cache and more batching, and it often speeds decode too. Validate quality before shipping, since aggressive quantization can degrade output.
• Separate interactive and batch traffic. Do not force a nightly document-processing job and a live chat to share the same tuning. Give the offline job a large-batch, latency-relaxed configuration and let the interactive endpoint run leaner. One size fits neither.
• Scale horizontally when a single replica saturates. Once a GPU is at its efficient operating point, more traffic means more replicas behind a load balancer, not a bigger batch on one overloaded card.

Matching the Trade-off to Your Actual Workload

The right balance is not a universal constant; it is a property of what you are building. Decide where you sit before you touch a single config value.

Interactive, human-in-the-loop features live and die by latency. A coding assistant, a chatbot, a search box with an AI answer: here a user is watching the screen, and every extra 200 milliseconds of TTFT erodes the experience. You bias toward lower latency, accept a smaller batch, and pay a bit more per token for the privilege of feeling instant. Prioritize TTFT and streaming above all.

Bulk, machine-consuming workloads are the mirror image. Classifying a million support tickets overnight, generating embeddings for a corpus, summarizing a document warehouse: no human is waiting on any single response, so per-request latency is nearly irrelevant. You crank batch size, maximize aggregate tokens per second, and drive cost per token to the floor. Prioritize throughput and cost, and let individual requests take as long as they need.

Most real products are a blend, and the mistake is serving both patterns from one undifferentiated endpoint tuned for neither. A useful mental model:

1. Classify each traffic source as latency-sensitive or throughput-sensitive before writing any config.
2. Give each class its own endpoint or its own tuning profile, even on the same hardware.
3. Measure p50 and p99 latency plus aggregate tokens per second for each class separately.
4. Adjust only the lever that moves the metric that class actually cares about.
5. Re-measure after every change, because LLM performance is deeply non-linear and intuition is unreliable here.

Follow that loop and you stop guessing. You will know, with numbers, that your chat endpoint holds a p95 TTFT under half a second while your batch pipeline chews through tokens at the hardware ceiling, and you will know exactly which knob to turn when either one drifts.

Bringing It All Together

Latency and throughput are not two names for speed. Latency is the experience of one request, dominated by prefill for the first token and by decode for everything after. Throughput is the economics of many requests, dominated by how well batching amortizes the fixed cost of loading model weights across concurrent work. The two pull against each other because the batch that maximizes GPU utilization taxes the individual request, and the KV cache that lets you batch at all is the same memory that context length devours.

Once you internalize that tension, the whole discipline of serving LLMs snaps into focus. You measure percentiles instead of averages. You cut prompts and cache prefixes to win latency for free. You lean on continuous batching and paged attention to win throughput for free. And when a genuine trade-off remains, you resolve it by asking a single honest question: is a human waiting on this response, or not? Answer that per workload, tune the matching lever, and re-measure. That is the entire game.

If you want to go deeper on the systems thinking behind production LLM serving, prompt caching, evaluation, batching strategy, and the cost math that turns a working prototype into a sustainable product, that is exactly what the AI Engineering Roadmap course on teachyou.ai is built to teach. It walks you from first principles through the real performance and reliability decisions engineers make when they ship LLM features to actual users, so the next time you get that "it's slow" ticket, you will already know whether it is a latency problem or a throughput one, and precisely what to do about it.