Load Testing AI Endpoints

Load testing AI endpoints is not the same job as load testing a REST API that returns a JSON blob in 40 milliseconds. When you load test AI endpoints, you are measuring how a model server behaves under concurrent generation: time to first token, tokens per second per stream, queueing behavior when the GPU batch fills up, and how gracefully the service degrades when you push past its real capacity. A tool like k6 or Locust that only checks status codes and total response time will tell you the endpoint "passed" while every user waited eleven seconds for the first word to appear. This article covers the metrics that matter, working k6 and Locust scripts for both blocking and streaming AI endpoints, and the ramp patterns that let you find breaking points without burning your entire model budget in one afternoon.

Why Load Testing AI Endpoints Is Different From Regular API Load Testing

A traditional load test cares about request count, response time, and error rate. Those three numbers still matter for AI endpoints, but they hide the parts that actually break user experience.

Three structural differences change how you approach this:

• Latency is not fixed. A traditional endpoint responds in roughly the same time regardless of the request body. An LLM endpoint's latency scales with output length, so a request asking for a 50-token answer and a request asking for a 2000-token essay are not comparable load, even though they hit the same route.
• Responses stream. Most production AI endpoints use server-sent events (SSE) or a chunked transfer encoding so the UI can render tokens as they arrive. A load test that waits for the full response and measures one round trip time is blind to time-to-first-token (TTFT), which is the number users actually feel.
• Capacity is batched, not per-connection. Inference servers like vLLM, TGI, or a hosted model API batch multiple concurrent requests onto the same GPU. Throughput per request drops as concurrency rises in a way that has nothing to do with network congestion. This means your load test has to sweep across concurrency levels to find the batching sweet spot and the point where it collapses.

Because of these three things, load testing AI endpoints means building a harness around token-level events, not just HTTP status codes. The rest of this guide builds exactly that harness with k6 and Locust, two tools you likely already have in your CI toolbox.

Metrics That Actually Matter When Load Testing AI Endpoints

Before writing a single script, agree on what you are going to measure. These are the metrics worth capturing on every run:

• Time to first token (TTFT). The gap between sending the request and receiving the first streamed chunk. This is the number that determines whether an interface feels responsive.
• Inter-token latency (ITL). The average gap between subsequent tokens once streaming has started. A model that streams smoothly at 40 tokens per second feels different from one that stutters, even if the total time is the same.
• Tokens per second (throughput). Aggregate across all concurrent streams, not per request. This tells you the real capacity of the server, not just one user's experience.
• p95 and p99 latency for both TTFT and total completion time. Averages hide the tail, and the tail is what shows up in support tickets.
• Error rate by type. Separate timeouts, 429 rate-limit responses, 5xx server errors, and truncated or malformed streams. They point to different root causes.
• Queue depth or wait time, if your inference server exposes it (vLLM and TGI both expose Prometheus metrics for this). A rising queue at constant load is the earliest signal you are near saturation.
• Cost per test run. Token-based billing means a load test itself has a dollar cost. Track total input and output tokens consumed so you can budget test runs the same way you budget staging infrastructure.

Write these down in your test plan before you run anything. A load test that only reports "average response time: 1.2s" for a streaming chat endpoint is not measuring the thing your users care about.

Setting Up k6 For AI Endpoint Load Testing

k6 is a solid default for AI load testing because it is scriptable in JavaScript, has native support for custom metrics, and handles high concurrency without needing a cluster of worker machines for moderate loads. Start with a non-streaming endpoint to validate the basics.

import http from 'k6/http';
import { check } from 'k6';
import { Trend, Counter } from 'k6/metrics';

const completionLatency = new Trend('completion_latency', true);
const rateLimitErrors = new Counter('rate_limit_errors');

export const options = {
  scenarios: {
    ramping_load: {
      executor: 'ramping-vus',
      startVUs: 1,
      stages: [
        { duration: '30s', target: 10 },
        { duration: '1m', target: 25 },
        { duration: '1m', target: 50 },
        { duration: '30s', target: 0 },
      ],
    },
  },
  thresholds: {
    completion_latency: ['p(95)<8000'],
    http_req_failed: ['rate<0.02'],
  },
};

export default function () {
  const payload = JSON.stringify({
    model: 'your-model-id',
    messages: [{ role: 'user', content: 'Summarize the benefits of caching in three bullet points.' }],
    max_tokens: 200,
  });

  const params = {
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${__ENV.API_KEY}`,
    },
  };

  const start = Date.now();
  const res = http.post(`${__ENV.BASE_URL}/v1/chat/completions`, payload, params);
  completionLatency.add(Date.now() - start);

  if (res.status === 429) {
    rateLimitErrors.add(1);
  }

  check(res, {
    'status is 200': (r) => r.status === 200,
    'has completion content': (r) => {
      try {
        const body = JSON.parse(r.body);
        return body.choices && body.choices[0].message.content.length > 0;
      } catch (e) {
        return false;
      }
    },
  });
}

Run it with an explicit VU ramp so you can correlate concurrency with degradation:

k6 run --env BASE_URL=https://staging.example.com --env API_KEY=$STAGING_KEY load_test.js

The ramping-vus executor is the key choice here. Instead of firing a fixed number of requests, it steps concurrency up in stages so you can see exactly where latency and error rate start climbing, rather than getting one aggregate number that averages the good part of the run with the bad part.

Load Testing Streaming Endpoints (SSE and WebSockets)

Most chat and completion endpoints in production stream. k6's core HTTP module does not parse SSE chunks natively for token-level timing, so the common pattern is to use k6's experimental streams support or drop to a small Node or Python harness for the token-timing portion, then feed the aggregated numbers back into your k6 dashboard as custom metrics.

Here is a Locust-based approach in Python, which handles streaming responses more naturally because you get a real HTTP client with chunk-level access:

import time
import json
from locust import HttpUser, task, between, events

class TokenTimingUser(HttpUser):
    wait_time = between(1, 3)

    @task
    def stream_completion(self):
        payload = {
            "model": "your-model-id",
            "messages": [{"role": "user", "content": "Explain connection pooling."}],
            "max_tokens": 300,
            "stream": True,
        }
        headers = {"Content-Type": "application/json"}

        start = time.perf_counter()
        first_token_time = None
        token_count = 0

        with self.client.post(
            "/v1/chat/completions",
            json=payload,
            headers=headers,
            stream=True,
            catch_response=True,
        ) as response:
            for line in response.iter_lines():
                if not line:
                    continue
                if first_token_time is None:
                    first_token_time = time.perf_counter()
                    ttft_ms = (first_token_time - start) * 1000
                    events.request.fire(
                        request_type="TTFT",
                        name="chat_stream_first_token",
                        response_time=ttft_ms,
                        response_length=0,
                        exception=None,
                    )
                token_count += 1

            total_ms = (time.perf_counter() - start) * 1000
            events.request.fire(
                request_type="TOTAL",
                name="chat_stream_complete",
                response_time=total_ms,
                response_length=token_count,
                exception=None,
            )
            response.success()

Run it with the standard Locust CLI, pointed at your staging or dedicated load-testing environment:

locust -f stream_test.py --host https://staging.example.com --users 50 --spawn-rate 5 --run-time 5m --headless --csv results

The --csv flag writes per-second aggregates you can pull into a spreadsheet or a Grafana panel afterward. The important part of this script is the first_token_time capture inside the iter_lines() loop. That single timestamp is what separates an AI-aware load test from a generic HTTP load test, because it is the only place you actually observe TTFT rather than inferring it from total response time.

Testing Rate Limits and Backpressure

Every production AI endpoint, whether you host the model yourself or call a hosted API, has a concurrency or token-per-minute ceiling somewhere. Your load test should deliberately go past that ceiling to confirm the system fails the way you expect: a 429 with a Retry-After header, not a hung connection or a 500 that leaks a partially generated response into your database.

A quick way to probe this with Vegeta, a minimal HTTP load tool that is easy to script in bash:

echo "POST https://staging.example.com/v1/chat/completions" | \
  vegeta attack \
  -body payload.json \
  -header "Authorization: Bearer $STAGING_KEY" \
  -header "Content-Type: application/json" \
  -rate 20/1s \
  -duration 60s | \
  vegeta report -type=json > results.json

cat results.json | jq '.status_codes'

Sweep -rate across a few values (5/1s, 20/1s, 50/1s) and watch how the status_codes breakdown shifts. What you want to see is a clean transition from mostly-200 to a controlled share of 429s as you cross the limit, with latency on the successful requests staying roughly flat. What you do not want to see is p99 latency climbing into the tens of seconds while the status code stays 200, because that means requests are queueing silently instead of being rejected, and users are sitting on a spinner with no feedback.

If your service sits behind an API gateway or a queue (SQS, a Redis-backed job queue, or similar), also test what happens when the queue itself backs up. A load test that only exercises the HTTP layer will miss a queue that grows unbounded and eventually OOMs the worker process.

Ramp-Up, Soak, and Spike Test Patterns for AI Systems

Three test shapes cover most of what you need before a launch or a scaling decision:

• Ramp-up test. Step concurrency from 1 to your expected peak over 5 to 10 minutes, as in the k6 example above. Use this to find the concurrency level where p95 TTFT crosses your SLA, not just where errors start.
• Soak test. Hold a moderate, sustainable concurrency (something below the point where you saw degradation in the ramp test) for 30 to 60 minutes. This surfaces memory leaks in the inference server, KV-cache fragmentation, and connection pool exhaustion that only show up after sustained load, not in a five-minute burst.
• Spike test. Jump from near-zero to 3x or 5x your expected peak in under a minute, hold for a couple of minutes, then drop back down. This mirrors what actually happens when a feature ships on Product Hunt or a marketing email goes out, and it tells you whether your autoscaler (if you have one) reacts fast enough, or whether the first wave of users eats a wall of timeouts while new GPU instances warm up.

Run these three in sequence against a staging environment that mirrors production instance types. GPU memory bandwidth and batching behavior do not transfer across instance families, so a load test result from a smaller GPU tier will not predict production behavior accurately.

Controlling Cost While Load Testing AI Endpoints

Every generated token during a load test is a token you are paying for, whether that is a hosted API bill or GPU-hours on your own inference cluster. A few habits keep this from getting expensive:

• Cap `max_tokens` aggressively during exploratory runs. You can find your concurrency breaking point with 50-to-100-token completions just as reliably as with 1000-token ones, and at a fraction of the cost.
• Use a dedicated staging deployment with its own budget alerts, not your production API key. This also protects production users from being crowded out by your own test traffic.
• Prefer a local model for high-volume repeat testing. Running the same load test script against a self-hosted vLLM or Ollama instance with a smaller open model lets you iterate on the test script itself for free, then run the final validated version against the real target once.
• Log token counts per test run the same way you log request counts. A dashboard that shows "12,000 requests, 1.4M tokens consumed, $X spend" per test run turns load testing into a planned line item instead of a surprise on the monthly invoice.

Mocking the Model vs Testing Against the Real Thing

There is a real tradeoff between testing against a mocked model response and testing against the live inference stack, and most teams need both.

A mock server that echoes back a fixed streamed response at a fixed token rate is useful for testing everything upstream of the model: your API gateway, your rate limiter, your connection handling, your client-side streaming UI. It is fast, free, and deterministic, which makes it good for catching regressions in your own infrastructure code in CI on every pull request.

It cannot tell you anything about GPU batching behavior, real TTFT under contention, or how your inference server behaves at 90 percent memory utilization. For that you need scheduled load tests against a real staging deployment, ideally on a cadence (weekly, or before any model or infrastructure change) rather than only on demand, since inference performance characteristics shift with model updates, driver updates, and even changes in average prompt length from real usage patterns.

A practical split: run the mocked version of your load test suite in CI on every merge to catch infrastructure regressions in minutes, and run the full real-model version on a schedule against staging, gated behind a manual approval if the token cost is significant.

Wiring Load Tests Into CI/CD

Once your k6 or Locust script is stable, treat it like any other automated test with a pass/fail gate, not just a manual exploration tool.

For k6, the thresholds block shown earlier already does this: a run exits non-zero if completion_latency p95 exceeds your SLA or if http_req_failed crosses your error budget. Wire that into a pipeline step:

load_test:
  stage: test
  script:
    - k6 run --env BASE_URL=$STAGING_URL --env API_KEY=$STAGING_KEY load_test.js
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
    - if: '$CI_MERGE_REQUEST_LABELS =~ /load-test/'

Gate the heavier, real-model version behind a schedule or a label so it does not run on every commit, and keep a lightweight mocked-endpoint version running on every pull request for fast feedback. Store the CSV or JSON output from each run as a pipeline artifact so you can trend TTFT and throughput over weeks, which is how you catch slow regressions (a model upgrade that quietly adds 200ms to TTFT) before they become a user complaint.

FAQ

What is the difference between load testing AI endpoints and load testing a normal API? Load testing AI endpoints requires measuring token-level metrics like time to first token and inter-token latency, not just total response time, because responses stream and latency scales with output length rather than staying roughly constant across requests.

Which tool is better for load testing AI endpoints, k6 or Locust? Both work well. k6 is easier to wire into CI with built-in thresholds and a smaller footprint per virtual user, which suits ramp-up and spike tests at scale. Locust's Python-based scripting makes it more natural to parse streaming responses and capture custom timestamps like TTFT, which suits detailed streaming analysis.

How do I measure time to first token in a load test? Send the request with streaming enabled, start a timer immediately before the request, and record the timestamp when the first chunk of the response body arrives, before you read the rest of the stream. Report that delta as a distinct metric from total completion time.

How much should I spend on load testing AI endpoints? Keep exploratory runs cheap by capping max_tokens and testing against a self-hosted or smaller model first, then run a final validated test against the real target endpoint with realistic prompt and output lengths. Track total tokens consumed per test run so the cost is visible rather than hidden in a monthly bill.

Should load tests run against production or a separate staging environment? Use a dedicated staging deployment with production-equivalent instance types and its own API key or quota. Testing directly against production risks starving real users of capacity and makes it hard to separate test traffic from real traffic in your metrics.

What is a good first concurrency target to test? Start well below your expected peak, ramp in small steps, and stop increasing once p95 time to first token crosses your SLA or the error rate exceeds your budget. That crossing point, not an arbitrary round number, is your real capacity ceiling for that instance configuration.