LangGraph Scaling: Running Many Graph Instances Concurrently

Your LangGraph agent works beautifully in the notebook. One user, one conversation, one graph execution at a time, and everything hums along. Then you ship it, fifty users show up on a Tuesday afternoon, and suddenly you are staring at connection pool errors, rate limit exceptions, and a checkpointer that has become the slowest component in your entire stack. LangGraph scaling is one of those topics that almost nobody thinks about until it hurts, because the framework makes single-run development so pleasant that it is easy to forget you are building something that eventually has to serve a crowd. The good news: LangGraph was designed with concurrency in mind, and the patterns for running many graph instances at once are well understood. The catch is that most of them are not obvious from the getting-started docs. In this article we will walk through how LangGraph actually executes concurrent runs, why "many graph instances" is usually the wrong mental model, and the specific engineering decisions — async invocation, checkpointer configuration, backpressure, horizontal workers — that separate a demo from a production deployment.

Why "Many Graph Instances" Is Really "Many Threads"

The first thing to internalize about LangGraph scaling is that you almost never want many graph instances. You want one compiled graph and many concurrent executions of it.

When you call builder.compile(), LangGraph produces a CompiledStateGraph object. This object is expensive-ish to build (it validates the graph structure, wires up channels, binds the checkpointer) but it is also stateless with respect to any individual conversation. All of the per-conversation state — the messages, the accumulated values in your state schema, the position in the graph — lives in the checkpoint, not in the compiled graph object. The compiled graph is a program; the checkpoint is the memory of a particular run of that program.

This means the compiled graph is safe to share. You compile it once at application startup, hold it in a module-level variable or an app-state container, and then invoke it concurrently from as many requests as you like. Each invocation carries a config with a thread_id, and that thread ID is what isolates one conversation from another:

from langgraph.graph import StateGraph, MessagesState, START
from langgraph.checkpoint.memory import InMemorySaver

builder = StateGraph(MessagesState)
builder.add_node("agent", call_model)
builder.add_edge(START, "agent")

# Compile ONCE at startup — this object is shared across all requests
graph = builder.compile(checkpointer=InMemorySaver())

# Each user/conversation gets its own thread_id
config_user_a = {"configurable": {"thread_id": "user-a-session-1"}}
config_user_b = {"configurable": {"thread_id": "user-b-session-7"}}

Developers coming from object-oriented backgrounds often reach for a factory that builds a fresh graph per request. Resist that instinct. Rebuilding and recompiling the graph on every request wastes CPU, defeats any caching the runtime does internally, and — if you also construct a fresh checkpointer each time — can silently break persistence, because an InMemorySaver created per request forgets everything the moment the request ends. One graph, many threads. That is the foundation everything else in this article builds on.

The exception worth noting: if different tenants genuinely need different graph topologies (different nodes, different tools), you can maintain a small registry of compiled graphs keyed by configuration, built lazily and cached. But even then, each distinct topology is compiled once and shared.

Async Execution: The Foundation of Concurrent Runs

LangGraph exposes both synchronous (invoke, stream) and asynchronous (ainvoke, astream) execution APIs. For any serious concurrency, you want the async path, and you want it end to end.

The reason is the nature of the workload. A graph execution spends the overwhelming majority of its wall-clock time waiting: waiting for the LLM API to return tokens, waiting for a tool to hit an external service, waiting for the checkpointer to write to the database. These are I/O waits, and Python's asyncio is extremely good at multiplexing thousands of I/O waits onto a single thread. A synchronous invoke inside a thread pool, by contrast, burns an OS thread for the entire duration of each run, and thread pools get expensive and awkward well before you reach hundreds of concurrent runs.

Here is the shape of fanning out many concurrent executions over a shared graph:

import asyncio

async def run_conversation(graph, thread_id: str, user_input: str):
    config = {"configurable": {"thread_id": thread_id}}
    result = await graph.ainvoke(
        {"messages": [{"role": "user", "content": user_input}]},
        config=config,
    )
    return result["messages"][-1].content

async def main():
    tasks = [
        run_conversation(graph, f"thread-{i}", prompts[i])
        for i in range(200)
    ]
    results = await asyncio.gather(*tasks, return_exceptions=True)

Two details matter here. First, return_exceptions=True keeps one failed run from cancelling the other 199 — at scale, individual failures are routine, and you want to handle them per-run rather than letting them propagate as a batch abort. Second, this only delivers real concurrency if everything underneath is async too: your model client, your tool functions, your checkpointer. A single synchronous requests.get() inside a node will block the entire event loop and stall every other concurrent run while it waits. Audit your nodes. Every network call inside a node should be await-ed, and every node that does I/O should be defined async def. If you are stuck with a synchronous library, wrap the call in asyncio.to_thread() so it at least moves off the event loop.

Within a single run, LangGraph also parallelizes for you: nodes that are triggered by the same super-step (fan-out edges) execute concurrently. That is intra-run parallelism. This article is about inter-run parallelism, but the two compose — a graph with parallel branches, run two hundred times concurrently, generates a lot of simultaneous I/O, which is exactly why the next two sections exist.

Checkpointers Under Load: From InMemorySaver to Postgres Pools

The checkpointer is where LangGraph scaling problems most often surface, because it sits on the write path of every super-step. After each node completes, LangGraph persists a checkpoint. Many concurrent runs means many concurrent writes, and your checkpointer configuration determines whether that is a non-event or a bottleneck.

InMemorySaver is fine for development and for genuinely ephemeral workloads, but it has two disqualifying properties for production: state vanishes on process restart, and it cannot be shared across multiple worker processes or machines. The moment you run more than one replica of your service, in-memory checkpoints mean a user's second request can land on a worker that has never heard of their conversation.

The standard production answer is AsyncPostgresSaver (or the SQLite variant for single-node, low-traffic cases). The critical configuration detail is the connection pool:

from psycopg_pool import AsyncConnectionPool
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver

pool = AsyncConnectionPool(
    conninfo="postgresql://user:pass@db-host:5432/agents",
    max_size=20,          # tune against your DB's max_connections
    kwargs={"autocommit": True, "prepare_threshold": 0},
)

checkpointer = AsyncPostgresSaver(pool)
await checkpointer.setup()   # creates tables; run once at startup

graph = builder.compile(checkpointer=checkpointer)

Sizing the pool is a balancing act. Too small, and concurrent runs queue up waiting for a connection — you will see graph executions mysteriously stall between nodes. Too large, and you exhaust Postgres's max_connections, especially once you multiply by the number of worker processes. If you run eight workers each with a pool of twenty, that is one hundred sixty potential connections before any other service touches the database. For deployments beyond a handful of workers, put a server-side pooler like PgBouncer in front of Postgres and keep per-worker pools modest.

Also think about what you are checkpointing. Every value in your state schema gets serialized on every checkpoint write. If your state carries a large document, a base64 image, or a bulky retrieval result, you are serializing and writing that payload after every single node. Store large artifacts in object storage or LangGraph's store interface and keep references — an ID, a URL — in graph state. Checkpoint payload size is one of the highest-leverage knobs in the whole system, and it is entirely under your control.

Backpressure: Semaphores, Queues, and Refusing Work Gracefully

asyncio.gather over two hundred tasks works, but "start everything at once" is not a scaling strategy — it is a load spike. Real systems need backpressure: a mechanism that limits how many graph runs execute simultaneously and decides what happens to the excess.

The simplest effective tool is a semaphore:

import asyncio

MAX_CONCURRENT_RUNS = 50
run_slots = asyncio.Semaphore(MAX_CONCURRENT_RUNS)

async def run_with_backpressure(graph, thread_id: str, payload: dict):
    async with run_slots:
        config = {"configurable": {"thread_id": thread_id}}
        return await graph.ainvoke(payload, config=config)

Fifty is not a magic number. The right cap is derived from your slowest shared resource. If your LLM provider allows a certain number of requests per minute and your average run makes four model calls, the arithmetic tells you how many runs per minute you can sustain, and the semaphore should keep in-flight runs at a level where you do not blow through that. The same logic applies to your database pool and to any rate-limited tool your agents call.

For request-driven services, pair the semaphore with a bounded queue and an explicit rejection path. When the queue is full, return an HTTP 429 or a "system busy" message immediately rather than accepting work you will process minutes later. Users tolerate a fast "try again shortly" far better than a request that hangs for ninety seconds and then times out. Unbounded acceptance is how you turn a traffic spike into a cascading failure: memory fills with pending state, timeouts fire mid-graph, retries pile onto the original load, and the system that would have degraded gracefully instead falls over completely.

Set timeouts at the run level too. Wrap ainvoke in asyncio.wait_for with a ceiling that makes sense for your product — a customer-facing chat turn probably should not run for five minutes no matter what the agent thinks it is doing. Because LangGraph checkpoints after every node, a timed-out run is not lost work: the thread can be resumed later from its last checkpoint, which is a genuinely underrated superpower for building resilient systems.

Surviving LLM Rate Limits Across Hundreds of Runs

Concurrent graph executions all funnel into the same model provider, and the provider does not care how elegant your graph is — it cares about requests per minute and tokens per minute. Rate limit handling is therefore a first-class part of LangGraph scaling, not an afterthought.

There are three layers to get right. The first is client-level rate limiting. LangChain chat model integrations accept a rate limiter, such as InMemoryRateLimiter, which smooths request bursts by making callers wait for a token-bucket slot before issuing a request. Because all your concurrent runs share the same model client instance (share it, just like the graph), a single limiter naturally coordinates across every run in the process.

The second layer is retry with exponential backoff. Rate limit responses (HTTP 429) are transient by definition; the correct response is to wait and retry, with jitter so that a hundred simultaneous retries do not re-collide. Most model clients support configurable retries, and you can add retry_policy on LangGraph nodes to retry a failed node without restarting the whole run — again leaning on the checkpoint system, since prior nodes' work is already persisted.

The third layer is architectural: reduce calls per run. Cache tool results that repeat across runs. Use cheaper, faster models for classification and routing nodes, reserving your expensive model for the nodes that need it. Batch operations where the workload allows. Every model call you eliminate is concurrency headroom you get for free.

One subtle trap: multiple worker processes each with their own in-memory rate limiter will collectively exceed the provider limit, because each process thinks it has the full budget. Once you scale horizontally, either divide the budget statically across workers or move rate limiting to a shared layer — a Redis-based token bucket, or an internal gateway that all workers route model traffic through.

Horizontal Scaling: Stateless Workers Behind a Shared Checkpointer

A single Python process, even fully async, eventually hits a ceiling — CPU-bound serialization work, a very large number of concurrent sockets, or simply the blast radius of one process crashing. The next step is horizontal scaling: multiple identical worker processes or containers, each holding the same compiled graph, all pointing at the same Postgres checkpointer.

The architecture that makes this work is precisely the one we set up earlier: workers are stateless, conversations live in the database. Because every super-step is checkpointed, any worker can pick up any thread. Request one for thread-42 can land on worker A, and request two for the same thread can land on worker C; worker C loads the checkpoint, sees the full conversation state, and continues as if nothing happened. There is no session affinity requirement, which keeps your load balancer configuration trivial.

For interactive chat workloads, a standard setup is FastAPI plus Uvicorn workers behind a load balancer, with the graph compiled in each worker's startup hook. For background and batch workloads — "run this research agent over ten thousand documents" — a task queue fits better: enqueue one job per graph run into Redis or a proper message broker, and have a fleet of consumers pull jobs, execute ainvoke, and acknowledge on completion. The queue gives you durable backpressure (jobs wait in the broker, not in memory), automatic retry on worker death, and easy elasticity — scale consumers up for the nightly batch, scale them down after.

Two coordination issues appear at this layer. First, concurrent writes to the same thread: if two requests for the same thread_id execute simultaneously on different workers, they will interleave checkpoint writes and produce a confused conversation. Guard against it with a per-thread lock — a Redis lock keyed on thread ID, or an application-level rule that a thread's requests are processed serially. Most chat UIs enforce this naturally (a user sends one message at a time), but API-driven and multi-agent setups need the explicit guard. Second, the shared rate limit budget discussed above. Neither problem is hard; both are much easier to design in than to retrofit after a production incident.

It is also worth knowing that LangGraph Platform, the managed offering, packages this whole pattern — queue, workers, Postgres persistence, horizontal autoscaling — as a service. Whether you buy or build, the underlying architecture is the same, and understanding it makes you better at operating either.

Keeping State Lean: The Hidden Cost Multiplier

Every scaling problem in LangGraph gets multiplied by the size of your state, so it deserves its own discussion. State is serialized on every checkpoint write, deserialized on every resume, and shipped over the wire to your database on both. A state object that is ten times larger makes your persistence layer roughly ten times more expensive at the same request rate.

The biggest offender in practice is unbounded message history. The default MessagesState happily accumulates every message forever, and long-running threads grow linearly. Past a point, this hurts twice: once in checkpoint size, and again in token cost, because you are stuffing the entire history into every model call. The remedies are standard but need to be deliberately applied — trim messages to a recent window before each model call, summarize older history into a compact rolling summary node, or both. LangGraph's message-trimming utilities and a periodic summarization node handle the common cases with little code.

Beyond messages, apply a simple rule: graph state holds coordination data, not payload data. IDs, statuses, short strings, small lists — fine. Full documents, embeddings, images, raw API responses — store them elsewhere and reference them. A retrieval node that writes forty full documents into state so a later node can read three of them is paying serialization tax on all forty at every subsequent checkpoint.

Finally, watch your reducers. Custom reducers that append without bound have the same growth pathology as message history. If a list in your state schema can grow with every node execution, decide its maximum useful size and enforce it in the reducer itself. Lean state is not an optimization you do at the end; it is a design habit that keeps every other part of the scaling story cheap.

Observability: Knowing What Your Fleet of Runs Is Doing

Running one graph, you can read the trace. Running five hundred concurrently, you need aggregates, and you need them before things break, because concurrent systems fail in ways that are invisible at low volume.

The metrics that matter for LangGraph scaling are: in-flight run count (against your semaphore cap), run duration percentiles (p50 tells you the normal case, p99 tells you what your unluckiest users experience), checkpoint write latency (your early-warning signal for database pressure), model call latency and 429 rate (your early-warning signal for provider pressure), and per-run error rate broken down by node. That last one is gold — when failures cluster on a single node, you have a specific tool integration or prompt to fix rather than a vague "the agent is flaky" report.

LangSmith gives you per-run tracing with node-level spans out of the box and is the natural first stop. Complement it with plain operational metrics from your own stack — Prometheus counters around your ainvoke wrapper, queue depth gauges, database pool utilization. The combination lets you answer both "what happened inside run X" and "what is the system doing right now," and you need both to operate concurrent agent workloads with confidence.

Instrument the recovery path too. Because checkpoints persist after every node, a crashed or timed-out run can be resumed by invoking the graph again on the same thread ID. At scale, build a small sweeper: find threads whose last checkpoint is older than a threshold and whose run never completed, and either resume them or mark them failed. Silent zombie threads are the concurrent-agent equivalent of leaked file handles — individually harmless, collectively a mess.

From One Run to Ten Thousand: A Practical Checklist

Pulling it all together, here is the path from notebook to production concurrency, in the order the problems will actually bite you.

1. Compile the graph once at startup and share it; isolate conversations with thread_id, never with separate graph instances.
2. Go fully async: ainvoke/astream, async nodes, async model clients, and no synchronous I/O anywhere on the event loop.
3. Replace InMemorySaver with AsyncPostgresSaver on a properly sized connection pool, and call setup() once at deploy time.
4. Add a semaphore-based concurrency cap derived from your slowest shared resource, plus per-run timeouts via asyncio.wait_for.
5. Layer rate limit defenses: a shared client-level rate limiter, node-level retry policies with backoff, and fewer model calls per run.
6. Keep state lean — trim or summarize message history, store payloads outside state, bound your reducers.
7. Scale out with stateless workers over the shared checkpointer, adding a per-thread lock and a shared rate limit budget.
8. Instrument in-flight counts, duration percentiles, checkpoint latency, and per-node error rates before you need them.

None of these steps is exotic, and that is the real lesson of LangGraph scaling: the framework's checkpoint-centric design means the hard distributed-systems problems — resumability, worker statelessness, crash recovery — are largely solved for you. Your job is the honest engineering around it: async discipline, connection management, backpressure, and restraint about what you put in state. Teams that treat those as day-one design constraints ship agents that shrug off traffic spikes; teams that discover them in production write very long incident reports.

If you want to go deeper — building graphs from scratch, mastering checkpointers and state design, and deploying agents that hold up under real traffic — our LangGraph Tutorial course on teachyou.ai walks through all of it hands-on, from your first node to a horizontally scaled deployment. The gap between a working agent and a scalable one is smaller than it looks, once you know exactly where to put the effort.