Batch Inference for LLMs

LLM batch inference means sending many prompts to a model as one job instead of one request at a time, trading immediate latency for much higher throughput and lower cost. You reach for it when the work is not interactive: nightly classification of support tickets, embedding a million documents, backfilling summaries, grading a dataset with an LLM judge, or generating product descriptions for a whole catalog. This article covers the two shapes batch inference takes in production (hosted batch APIs and self-hosted continuous batching), when each one wins, and how to run both with real code.

Why LLM batch inference exists

A single LLM request wastes the hardware it runs on. The model does a small amount of math per token, and a GPU can do far more math in parallel than one request needs. When you serialize requests, the accelerator sits mostly idle waiting on memory and network. LLM batch inference fixes this by packing many sequences through the model together, so the same forward pass produces tokens for dozens or hundreds of prompts at once.

There are two independent reasons to batch, and they map to two different tools:

• Cost. Hosted providers offer a discounted batch tier because your work can be scheduled off-peak and packed efficiently on their side. You submit a file of requests, wait, and download results. Latency is measured in minutes to hours, not milliseconds.
• Throughput on your own hardware. If you run open-weight models on your own GPUs, batching is how you turn a single card into a service that handles real volume. Here the batching happens continuously inside the serving engine.

Confusing these two is the most common mistake. "Batch inference" on a provider means a specific async API with a discount. "Batching" inside vLLM or TensorRT-LLM means the scheduler packing concurrent requests. You often use both: continuous batching for live traffic, and a hosted batch API for the giant offline jobs where you do not care about latency.

Hosted batch APIs: the cheapest path for offline work

If you already call a frontier model over an API, the fastest cost win is the batch endpoint. Both Anthropic and OpenAI expose one. The pattern is the same across providers: build a list of independent requests, submit them as a job, poll until done, then read results. The provider guarantees completion within a stated window and charges a lower per-token rate than the synchronous API. Do not quote yourself a specific discount number from memory; check the current pricing page, because it changes.

The important properties to design around:

• Requests in a batch are independent. There is no ordering guarantee and no shared context between items. Each request carries its own full prompt.
• You correlate inputs to outputs with a custom ID you attach to every request. Results come back keyed by that ID, not by position.
• Partial failure is normal. Some requests can error (a bad prompt, a content filter) while the rest succeed. Your reader must handle per-item status.

Here is a complete Anthropic Message Batches flow. It submits a batch, polls, and reconciles results by custom_id.

import time
from anthropic import Anthropic
from anthropic.types.messages.batch_create_params import Request
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming

client = Anthropic()

prompts = {
    "ticket-1001": "Classify this ticket as billing, bug, or feature: 'App crashes on export.'",
    "ticket-1002": "Classify this ticket as billing, bug, or feature: 'Charged twice this month.'",
    "ticket-1003": "Classify this ticket as billing, bug, or feature: 'Please add dark mode.'",
}

requests = [
    Request(
        custom_id=cid,
        params=MessageCreateParamsNonStreaming(
            model="claude-sonnet-4-5",
            max_tokens=64,
            messages=[{"role": "user", "content": text}],
        ),
    )
    for cid, text in prompts.items()
]

batch = client.messages.batches.create(requests=requests)
print("submitted", batch.id)

while True:
    batch = client.messages.batches.retrieve(batch.id)
    if batch.processing_status == "ended":
        break
    time.sleep(30)

for result in client.messages.batches.results(batch.id):
    cid = result.custom_id
    if result.result.type == "succeeded":
        text = result.result.message.content[0].text
        print(cid, "->", text.strip())
    else:
        print(cid, "FAILED", result.result.type)

Two things about this code matter in production. First, the poll loop is deliberately dumb: a fixed sleep and a status check. Do not busy-poll every second; you gain nothing and you add load. A 30 to 60 second interval is fine because the whole point is that you are not waiting synchronously. Second, the results iterator streams line-delimited JSON, so you never hold the whole result set in memory. For a million-item batch you write each result straight to a database or file as it arrives.

The OpenAI Batch API uses a file-based flow instead of an inline request list. You upload a JSONL file where each line is a request, create a batch pointing at that file, poll, then download an output file. The mental model is identical; only the plumbing differs.

import json, time
from openai import OpenAI

client = OpenAI()

rows = [
    {
        "custom_id": "ticket-1001",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4.1-mini",
            "max_tokens": 64,
            "messages": [{"role": "user", "content": "Classify: 'App crashes on export.'"}],
        },
    },
    # ... one object per line, thousands of them
]

with open("batch_input.jsonl", "w") as f:
    for r in rows:
        f.write(json.dumps(r) + "\n")

infile = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=infile.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

while True:
    batch = client.batches.retrieve(batch.id)
    if batch.status in ("completed", "failed", "expired", "cancelled"):
        break
    time.sleep(60)

if batch.status == "completed":
    out = client.files.content(batch.output_file_id).text
    for line in out.splitlines():
        obj = json.loads(line)
        cid = obj["custom_id"]
        body = obj["response"]["body"]
        print(cid, "->", body["choices"][0]["message"]["content"].strip())

The JSONL-per-line format is worth internalizing because it is the lingua franca of batch inference. vLLM, most eval harnesses, and every provider batch tool speak it. Keep your dataset in that shape from the start and you can move a job between a provider batch API and a local engine with almost no rewrite.

When hosted batch APIs win:

• The volume is large and the deadline is loose (hours, not seconds).
• You want the cheapest per-token rate without running any infrastructure.
• The model you want is closed-weight and only available via API.
• The work is embarrassingly parallel with no cross-request dependencies.

When they lose: anything interactive, anything where you need a result in the same request cycle, or work on models you must run yourself for privacy or licensing reasons. That is where self-hosted batching comes in.

Self-hosted LLM batch inference with vLLM

If you run open-weight models, vLLM is the default serving engine for batch inference, and the key idea it gives you is continuous batching. Understanding that idea is what separates a setup that gets 5x throughput from one that gets 50x.

Naive (static) batching groups N requests, runs them all until every one finishes, then starts the next group. The problem: generations have wildly different lengths. One prompt produces 20 tokens, another produces 2000. In a static batch, the whole batch is held hostage by the longest sequence, and the GPU slots that finished early sit idle. Continuous batching, also called in-flight batching, instead treats the batch as a living set: the moment one sequence finishes, its slot is freed and a waiting request is admitted on the very next decoding step. The batch is refilled continuously, so the GPU stays saturated.

Two subsystems make this work, and you tune both:

• The scheduler decides which requests are running versus queued each step. It is bounded by how many tokens of KV cache fit in GPU memory.
• PagedAttention manages the KV cache like virtual memory, in fixed-size blocks, so sequences of different lengths pack tightly without fragmentation. This is why vLLM can hold far more concurrent sequences than a naive allocator.

For a pure offline job (you have all prompts up front and just want them all processed as fast as possible), use the offline LLM class. It applies continuous batching internally across your whole prompt list.

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    max_num_seqs=256,          # max concurrent sequences in a batch
    gpu_memory_utilization=0.90,
)

sampling = SamplingParams(temperature=0.0, max_tokens=128)

prompts = [f"Summarize in one sentence: {doc}" for doc in load_documents()]

# vLLM schedules all of these with continuous batching; you just get a list back.
outputs = llm.generate(prompts, sampling)

for out in outputs:
    print(out.prompt[:40], "->", out.outputs[0].text.strip())

That single llm.generate call is doing the batching for you. You do not chunk the prompts yourself; feeding the whole list lets the scheduler keep the batch full. The three knobs that move throughput the most:

• max_num_seqs: the ceiling on concurrent sequences. Higher means more parallelism and more throughput, until you run out of KV cache and requests start queueing (which is fine) or you hit out-of-memory (which is not). Raise it until throughput stops improving.
• gpu_memory_utilization: how much of the card vLLM claims for weights plus KV cache. Push it toward 0.90 to 0.95 on a dedicated box to buy more cache, which means more concurrent sequences.
• max_num_batched_tokens: the token budget per scheduler step. This governs the balance between admitting new prompts (prefill) and generating tokens for running ones (decode).

For a service that also takes live requests, run vLLM as a server instead. It exposes an OpenAI-compatible endpoint and applies the same continuous batching across whatever concurrent requests arrive.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.90 \
  --max-num-batched-tokens 8192

Now any client that speaks the OpenAI Chat Completions format can hit it, and you get batching across concurrent callers without them knowing. To push a large offline dataset through this server, fire requests concurrently so the scheduler always has a full queue. A bounded async client is the standard tool.

import asyncio, json
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="none")
sem = asyncio.Semaphore(128)   # cap in-flight requests to match server capacity

async def one(row):
    async with sem:
        resp = await client.chat.completions.create(
            model="meta-llama/Llama-3.1-8B-Instruct",
            max_tokens=128,
            messages=[{"role": "user", "content": row["prompt"]}],
        )
    return row["id"], resp.choices[0].message.content

async def main():
    rows = [json.loads(l) for l in open("dataset.jsonl")]
    results = await asyncio.gather(*(one(r) for r in rows))
    with open("out.jsonl", "w") as f:
        for cid, text in results:
            f.write(json.dumps({"id": cid, "text": text}) + "\n")

asyncio.run(main())

The semaphore is the whole trick. Without it you either flood the server with tens of thousands of open sockets or you serialize and starve the batch. Set the concurrency limit near the server's max_num_seqs so the scheduler always has work but the queue does not grow unbounded.

Throughput versus latency: the tradeoff you are actually making

Every batching decision is a point on the same curve. Bigger batches mean higher throughput (more tokens per second across all users) and higher per-request latency (any single user waits longer). Smaller batches mean the opposite. Batch inference lives at the throughput end on purpose.

Concretely, watch these signals:

• Time to first token (TTFT). Grows as batch size grows, because prefill for many prompts competes. Irrelevant for offline batch, critical for interactive.
• Tokens per second, aggregate. This is what you maximize for batch jobs. It keeps climbing with concurrency until the KV cache is full, then plateaus.
• Queue depth. In a server, a healthy batch job keeps the queue non-empty (so the batch stays full) but not exploding (which just adds latency with no throughput gain).

A practical tuning loop for a self-hosted offline job: start with max_num_seqs at 128, run a representative slice, record tokens per second. Double it, run again. When throughput stops rising or you hit memory pressure, back off one step. You have found the knee of the curve. Do not tune on toy prompts; use real input and output lengths, because KV cache pressure is driven by sequence length, and short test prompts will lie to you.

Prompt design and cost for batch jobs

Batch inference amplifies both good and bad prompt decisions by your item count. A 200-token instruction preamble repeated across a million requests is 200 million input tokens you pay for every run. Two levers help:

• Prompt caching. If your requests share a large common prefix (a long system prompt, a fixed rubric, few-shot examples), put the shared part first and mark it cacheable. Providers and vLLM can reuse the computed prefix across requests so you pay full price for it far less often. Order matters: the cacheable, identical content must come before the per-item content.
• Structured output. For classification, extraction, or grading, constrain the model to a small schema (JSON with a fixed set of fields, or a single label). Fewer output tokens means lower cost and faster completion, and it makes the results trivial to parse downstream. vLLM supports guided decoding against a JSON schema or regex; the provider APIs support structured output modes. Use them for batch work, where you cannot eyeball each response.

One more cost note specific to hosted batch: the discount is real, but the failure and retry behavior is what actually determines your bill. If 3 percent of a huge batch fails on a transient error and you resubmit the whole batch, you pay for the 97 percent twice. Always resubmit only the failed custom_ids. Keep your input keyed by ID so a retry is a filter, not a rerun.

A decision checklist

Use this to pick a path quickly:

1. Is the work interactive (a user is waiting)? If yes, this is not batch inference. Serve it live, and let continuous batching in your engine handle concurrency.
2. Is it offline and the model is closed-weight (Claude, GPT)? Use the provider batch API. Cheapest, no infra.
3. Is it offline and the model is open-weight, with data or licensing reasons to self-host? Use vLLM offline LLM.generate on your own GPU.
4. Do you need both live serving and periodic large batches on the same open model? Run vLLM as a server, feed offline jobs through it with a bounded async client, and let one scheduler serve both.
5. Whichever path: key every request by a stable ID, stream results to storage, and retry only failures.

FAQ

What is the difference between batch inference and continuous batching? Batch inference is the goal: process many prompts as one job for throughput and cost, accepting higher latency. Continuous batching is one technique to achieve it inside a serving engine, where finished sequences are evicted and new ones admitted every decoding step to keep the GPU full. A hosted batch API gives you batch inference without you managing any batching; a self-hosted engine gives you batch inference by doing continuous batching under the hood.

When should I use a provider batch API versus running vLLM myself? Use the provider batch API when the model is closed-weight, the deadline is loose, and you want zero infrastructure and the discounted rate. Run vLLM yourself when you need an open-weight model, must keep data on your own hardware, want control over latency, or already own GPUs whose cost is fixed. Many teams use both: vLLM for live traffic, a provider batch API for occasional giant offline jobs.

How large should my batch be? For hosted APIs, batch as large as the provider allows per job and split by their documented limits; there is no latency penalty you care about. For self-hosted vLLM, do not set a batch size directly. Set max_num_seqs and gpu_memory_utilization, feed all your prompts, and let the scheduler decide. Tune max_num_seqs upward until throughput stops improving or memory runs out.

Does batch inference change the model's output? No. Batching is an execution and scheduling optimization; each request is computed independently and gets the same result it would alone, given the same sampling parameters. Set temperature=0 if you want determinism per item. What batching changes is timing and cost, not the tokens produced.

How do I handle failures in a batch job? Assume partial failure is normal. Attach a unique custom_id to every request, and when results come back, record status per item. Resubmit only the failed IDs, never the whole batch, so you do not pay twice for the parts that succeeded. Stream results to a database or file keyed by ID as they arrive rather than holding everything in memory.

Can I get structured JSON out of a batch job reliably? Yes, and you should. On vLLM use guided or structured decoding against a JSON schema or regex so every output parses. On provider APIs use their structured output mode. For offline batch work you cannot inspect responses by hand, so constraining the shape up front is what makes the pipeline trustworthy and cheap to parse.

Does prompt caching help batch inference? Yes, when your requests share a large common prefix such as a fixed system prompt, rubric, or few-shot examples. Put the shared, identical content first and mark it cacheable so the compute for that prefix is reused across items instead of paid for every request. It cuts both cost and time for jobs where the instruction dwarfs the per-item input.