Agent SLAs: Setting Realistic Uptime and Latency Expectations

The Meeting Where Someone Says "99.9% Uptime"

It happens in almost every project kickoff. A stakeholder who has spent years around traditional web services asks what the uptime guarantee is for the new agent, and someone confidently says "99.9%, same as our API." Everyone nods. Nobody in the room has actually load-tested an LLM-backed agent under production conditions yet.

Six weeks later, the on-call engineer is staring at a dashboard where the agent's p99 latency spikes to 40 seconds every time a downstream vector database gets slow, the model provider silently degrades quality during a regional incident, and a tool call retries three times before giving up, burning 90 seconds per request. The SLA everyone agreed to is already broken, and it was broken from day one because it was copied from a system with fundamentally different failure modes.

Agent SLAs are not the same discipline as web service SLAs. An agent is a pipeline: a prompt goes in, a model reasons over it, tools get called, results get parsed, sometimes another model call happens, and eventually something comes out the other end. Every link in that chain has its own latency distribution and its own failure rate, and they compound in ways that traditional service-level thinking does not anticipate. If you are building or operating agents in production, you need a different mental model for uptime and latency, one that is honest about where the variance actually comes from. This article walks through how to build that model, how to set numbers you can actually hit, and how to communicate them to stakeholders without either overpromising or scaring them into paralysis.

Why Agent SLAs Break the Old Playbook

Traditional SLA thinking assumes a few things that don't hold for agents. First, it assumes latency is roughly deterministic for a given load level — a database query takes X milliseconds, plus or minus some jitter from cache misses. Second, it assumes failures are binary and attributable — the service either responded correctly or it threw a 500, and you know which component caused it. Third, it assumes uptime is mostly about your own infrastructure staying alive.

None of these hold cleanly for agents.

Latency is not deterministic, it's a distribution shaped by reasoning depth. A simple factual query might resolve in two seconds. A query that requires the agent to call three tools, evaluate the results, decide it needs a fourth tool, and then synthesize an answer might take 25 seconds — and the agent doesn't know in advance which path it will take. The same prompt, run twice, can produce different tool-call sequences and therefore different latencies, because LLM output is not fully deterministic even at low temperature.

Failures are often partial and hard to attribute. Did the request fail because the model produced malformed JSON for a function call? Because a tool timed out? Because the model hallucinated a tool that doesn't exist? Because a rate limit kicked in upstream at the provider? Each of these needs a different remediation, but from the outside they can all look like "the agent didn't respond."

Uptime depends on infrastructure you don't control. Your agent's uptime is capped by your model provider's uptime, and providers do have incidents. If OpenAI, Anthropic, or any other provider has a regional outage or a degraded-service event, your agent inherits that outage no matter how well you've engineered everything else. This is fundamentally different from a service where you own the full stack down to the database.

Once you internalize these three differences, you stop trying to set a single "99.9% uptime, 200ms p95" SLA and start decomposing the problem into the pieces that actually determine what users experience.

Decomposing the Agent Pipeline into Measurable Stages

Before you can promise anything, you need to know where time and reliability actually go. Break the agent's execution into discrete stages and instrument each one separately.

• Request ingestion and queueing — time from the user's request arriving to the agent actually starting work. This is where backpressure and queue depth show up under load.
• Model inference (per call) — time for each individual LLM call, including any that happen mid-reasoning for tool selection or re-planning.
• Tool execution — time spent in external calls: database queries, API calls to third-party services, code execution sandboxes, retrieval from a vector store.
• Orchestration overhead — the glue code between model calls and tool calls: parsing function-call arguments, validating schemas, routing to the right tool handler.
• Response streaming and delivery — time from the final token being generated to it reaching the user, which matters a lot if you're not streaming.

Once you log timestamps at each boundary, you can build a real picture of where a typical request spends its time, and more importantly, where the outliers spend theirs. In most production agents we've profiled, the surprising finding isn't that model inference is slow — it's that tool execution and retry logic account for the majority of tail latency. A single flaky external API with a 30-second timeout, called even once in a chain of four tool calls, will dominate your p99 far more than any amount of prompt optimization.

import time
from contextlib import contextmanager

STAGE_TIMINGS = {}

@contextmanager
def measure_stage(request_id, stage_name):
    start = time.monotonic()
    try:
        yield
    finally:
        elapsed = time.monotonic() - start
        STAGE_TIMINGS.setdefault(request_id, {})[stage_name] = elapsed

# usage inside the agent loop
with measure_stage(req.id, "model_inference"):
    plan = model.generate(prompt)

with measure_stage(req.id, "tool_execution"):
    result = tool_registry.call(plan.tool_name, plan.args)

with measure_stage(req.id, "orchestration"):
    parsed = validate_and_parse(result)

This kind of instrumentation is unglamorous but it's the single highest-leverage thing you can do before setting any SLA number. You cannot promise what you haven't measured, and in agent systems the bottleneck is almost never where intuition says it will be.

Setting Latency Targets: Percentiles, Not Averages

Never publish an average latency as your SLA. Averages are meaningless for agent workloads because the distribution is heavily right-skewed — most requests are fast, a meaningful tail is very slow, and the average gets dragged around by that tail without telling anyone what to expect.

Instead, commit to percentiles, and commit to more than one:

1. p50 (median) — what a typical user experiences. This is your "normal day" number.
2. p90 — what most users experience even on a busier day. This is a good number to put in front of product teams.
3. p99 — your worst-case-but-not-catastrophic number. This is what you use for alerting thresholds.
4. Timeout ceiling — the hard cutoff where you give up and return a fallback response rather than let the user wait indefinitely.

A realistic latency SLA for a moderately complex agent (one or two tool calls, single model in the loop) in 2026 production conditions might look like: p50 under 4 seconds, p90 under 12 seconds, p99 under 30 seconds, hard timeout at 45 seconds with graceful degradation. Notice how wide that band is compared to a typical REST API SLA. That width isn't sloppiness — it's an honest reflection of variable reasoning depth and tool-call fan-out.

If your product genuinely cannot tolerate that variance — say, a voice agent where anything over 2 seconds feels broken — that's a signal you need architectural changes, not a tighter SLA slapped on top of the same architecture. Streaming partial responses, using a smaller/faster model for a first-pass acknowledgment while a larger model reasons in the background, or caching common query patterns are real levers. Wishing the p99 down without changing the system underneath it is not.

Setting Uptime Targets: Separate "Available" from "Correct"

Uptime for an agent needs to be split into at least two separate metrics, because they fail independently and mean different things to users.

Availability — is the agent responding at all within your timeout window? This is closer to the traditional definition: did the request get a response (of any quality) before the deadline.

Task success rate — of the responses that came back, how many actually accomplished what the user asked? An agent can be "available" 99.9% of the time and still fail the user constantly if it's returning malformed tool calls, giving up after retries, or answering the wrong question because a RAG lookup returned irrelevant context.

Most teams that get burned by agent SLAs made the mistake of only tracking availability. Their dashboards were green while user complaints piled up, because the thing that was breaking was correctness, not uptime. When you set your SLA, set both numbers, and be honest that they will likely differ. It's common to see 99.5% availability paired with something like 92% task success rate on complex multi-step tasks, and that gap is not a bug in your monitoring — it's an accurate reflection of how hard the underlying task is.

• Track availability as: (responses returned within timeout) / (total requests)
• Track task success as: (responses that passed automated eval or got a positive user signal) / (total responses returned)
• Track a combined "effective success rate" as availability × task success, which is the number that actually matters to the business

When you present this to a stakeholder who wants "99.9% uptime," walk them through this decomposition. It reframes the conversation from an unrealistic single number to a set of numbers that are each individually achievable and honestly tracked.

Building In Degradation Paths Before You Need Them

An SLA is not just a target, it's a commitment about what happens when the target is missed. If your agent has no fallback behavior, then every SLA breach is a hard user-facing failure. The fix is to design graceful degradation into the agent before you ever publish a number.

Some patterns that consistently work in production:

• Timeout-triggered fallback responses. If the full reasoning chain exceeds your p99 threshold, cut it off and return the best partial answer available, clearly labeled as partial, rather than letting the user wait indefinitely for a response that may never come.
• Tool-level circuit breakers. If a specific tool has failed the last N times or is trending toward its own timeout, stop calling it for a cooldown period and let the agent either skip that step or substitute a cached/approximate result.
• Model fallback tiers. Keep a faster, cheaper model on standby that can handle a simplified version of the task if your primary model is degraded or rate-limited. This won't match the primary model's quality, but a slightly worse answer within SLA usually beats a great answer that never arrives.
• Retry budgets, not infinite retries. Cap retries per request stage (for example, at most one retry on a tool call, at most one re-plan on a malformed function call) and track how often you exhaust that budget. Unbounded retries are one of the most common causes of runaway p99 latency in agent systems.

def call_with_fallback(primary_fn, fallback_fn, timeout_s, request_id):
    start = time.monotonic()
    try:
        return primary_fn(timeout=timeout_s)
    except (TimeoutError, ModelUnavailableError) as e:
        elapsed = time.monotonic() - start
        log_degradation_event(request_id, reason=str(e), elapsed=elapsed)
        return fallback_fn()

Design and test these paths under simulated failure before you publish your SLA. If your only answer to "what happens when the model provider has an incident" is "the agent is down," say that explicitly in your SLA rather than pretending it won't happen. A published SLA that acknowledges dependency risk is more trustworthy than one that silently assumes perfect upstream reliability.

Load Testing Agents: What's Different From Load Testing APIs

Standard load testing tools assume request/response pairs with roughly stable payload shapes. Agent load testing needs to account for a few things those tools don't handle well out of the box.

Vary prompt complexity in your test set. If every load test request is a trivial single-turn query, you'll never see the tail latency caused by multi-hop tool chains. Build a representative sample that mirrors your real traffic mix — a good rule of thumb is to pull actual anonymized production queries into your load test corpus rather than hand-writing synthetic ones, since hand-written test prompts tend to cluster around the easy cases.

Test concurrent tool contention, not just concurrent model calls. If ten agent instances all hit the same downstream database or the same rate-limited third-party API simultaneously, the bottleneck may be entirely outside your model provider. Load test the full path, including shared resources your tools depend on.

Measure cost alongside latency. Under load, some systems handle backpressure by silently routing to a more expensive model tier or retrying more aggressively, which fixes your latency numbers while quietly blowing up your per-request cost. Track cost per request as a first-class metric during load tests, not an afterthought.

Simulate provider degradation, not just outright outages. The failure mode that catches teams off guard most often isn't the model provider going fully down — it's the provider staying up but responding 3-5x slower than normal during a partial incident. Your circuit breakers and fallback tiers need to trigger on slow-but-alive responses, not just hard errors, so make sure your load tests include a "degraded but responding" scenario.

Communicating SLAs to Non-Technical Stakeholders

Getting the numbers right is only half the job. The other half is presenting them in a way that doesn't get flattened back into "99.9% uptime, we promise" by the time it reaches a sales deck or a customer contract.

A few practices that help:

1. Lead with the percentile band, not a single number. Instead of "sub-second response time," say "most responses in under 4 seconds, nearly all within 30 seconds, with a hard cutoff at 45 seconds where we return a partial result."
2. Separate the uptime conversation from the quality conversation explicitly. Stakeholders need to understand that "the agent responded" and "the agent got it right" are different guarantees, especially if the agent is doing anything with legal, medical, or financial stakes.
3. Name the upstream dependency risk out loud. If your uptime is capped by your model provider's own SLA, say so in plain terms: "our target assumes our model provider is operating normally; provider-side incidents are outside our control but we will fail over to [fallback] when detected."
4. Revisit the SLA on a fixed cadence. Model providers change latency characteristics with new releases, your own traffic mix shifts, and tool dependencies get added or removed. An SLA set once at launch and never revisited becomes fiction within a quarter.

The teams that get the best reception from stakeholders aren't the ones with the most aggressive-sounding numbers — they're the ones whose numbers hold up three months later because they were grounded in actual measured behavior rather than aspiration.

Common Mistakes Teams Make With Agent SLAs

A quick list of patterns worth avoiding, drawn from the failure modes above:

• Copying an SLA template from a traditional API service without re-deriving it from measured agent behavior.
• Publishing an average latency instead of a percentile distribution.
• Tracking only availability and ignoring task success rate, so the dashboard stays green while users are unhappy.
• Allowing unbounded retries on tool calls or re-plans, which turns a single flaky dependency into a runaway tail latency problem.
• Never testing what happens when the model provider is degraded rather than fully down.
• Setting a single global SLA for every task type, when a simple lookup and a five-step multi-tool workflow have wildly different realistic latency profiles and probably deserve separate SLAs.
• Failing to revisit the SLA after a model upgrade, a new tool integration, or a meaningful shift in traffic patterns.

Every one of these is fixable, and none of them require exotic infrastructure. They require discipline: instrument the pipeline, measure the real distribution, decompose uptime into availability and correctness, build degradation paths before you need them, and communicate honestly about what depends on someone else's infrastructure.

Where to Go Deeper

Setting realistic agent SLAs is ultimately a systems design skill, not a negotiating tactic. It requires understanding how orchestration, tool calling, retries, and model behavior interact under load, and it requires the engineering discipline to instrument every stage of the pipeline before making promises about any of them. This is exactly the kind of production-facing skill that separates agents that survive contact with real users from ones that look great in a demo and fall over in week two.

If you want to build this skill hands-on rather than just reading about it, our 30 Days of Hermes Agent course walks through building a production-grade agent from scratch, including the observability, retry logic, and degradation patterns discussed here, so that by the end you're not guessing at your SLA numbers, you're deriving them from an agent you built and load-tested yourself.