Setting Up LangSmith Alerts for Production LLM Monitoring

Your LLM app worked in the demo. It worked in staging. Then it shipped, real users started hitting it at 2am on a Tuesday, and three weeks later someone in a Slack channel asks why support tickets are up 40% with no obvious cause. You check the logs. There's nothing obviously wrong — no crashes, no 500s, no red dashboards. The model has just been quietly getting worse at its job, and nobody was watching for that because nobody set up the monitoring to catch it. This is the single most common failure mode in production LLM systems, and it's entirely preventable with LangSmith production monitoring done properly. This isn't a "turn on the dashboard and forget it" problem — it's an ops discipline, and it needs the same rigor you'd apply to any other production system, plus a few LLM-specific twists that traditional APM tools weren't built for.

This article walks through what to actually watch, how to set alert thresholds that don't cry wolf, how to route alerts so the right person gets woken up (or doesn't), and what a runbook should look like when an alert does fire. We'll use LangSmith as the concrete tool since it's what most teams building on LangChain or plain LLM API calls reach for, but the underlying principles apply regardless of which observability platform sits behind your traces.

Why LLM monitoring isn't just APM with extra steps

Traditional application monitoring assumes deterministic failure modes: a service is up or down, a request succeeds or throws, a query is fast or slow. LLM systems fail in a much stranger space. A request can return a 200 OK with a perfectly well-formed response that is nonetheless wrong, hallucinated, off-brand, or subtly worse than what you shipped last month — with no exception anywhere in the stack. Traditional monitoring is blind to this entire category of failure.

That's why production LLM monitoring needs at least two layers running simultaneously:

• Infrastructure-level signals — errors, latency, cost, token usage. These are the things you'd monitor for any API-backed service, adapted for LLM-specific failure patterns.
• Quality-level signals — is the model's output actually good, on an ongoing basis, not just at the moment you deployed it. This is the layer most teams skip, and it's the one that catches silent regressions.

LangSmith is useful here specifically because it gives you both in one place: it captures full traces (prompts, completions, tool calls, latency, token counts) and it lets you run evaluators against live traffic, not just against a static test set at deploy time. Setting this up properly is what separates teams that find out about a regression from a churned customer versus teams that find out from a dashboard.

Error rates: know the difference between "the API is down" and "the model refused"

Start with the boring stuff, because it's boring precisely because it's foundational. Track:

• HTTP/API failure rate — connection errors, 5xx from the model provider, rate limit rejections (429s). These usually mean an upstream outage or you've blown through a quota.
• Timeout rate — requests that never came back within your SLA window. With streaming responses this is subtler: a stream can start and then stall, which a naive timeout check on "time to first token" will miss entirely.
• Application-level failures — output parsing failures (the model didn't return valid JSON when your schema required it), tool-call failures, content filter rejections, refusals.

The mistake most teams make is lumping all of these into one "error rate" metric. Don't. A spike in 429s tells you to talk to your model provider about quota. A spike in JSON parsing failures tells you your prompt or schema needs work. A spike in refusals might mean users are hitting the model with content it won't touch, or it might mean the provider silently tightened their safety filters upstream of you. These need different people to look at them, so they need to be different metrics from the start, not one blended "errors" number you have to unpack after the fact.

In LangSmith, tag runs with an error type at the point of failure (catch the exception, classify it, attach it as run metadata) rather than relying on a generic "run failed" boolean. This is the single highest-leverage thing you can do to make error-rate alerts actionable instead of just noisy.

from langsmith import traceable
from langsmith.run_helpers import get_current_run_tree

@traceable(name="llm_call")
def call_model(prompt: str):
    run = get_current_run_tree()
    try:
        response = client.chat.completions.create(
            model="gpt-4.1",
            messages=[{"role": "user", "content": prompt}],
            timeout=15,
        )
        return response
    except RateLimitError as e:
        run.extra = {**(run.extra or {}), "error_type": "rate_limit"}
        raise
    except APITimeoutError as e:
        run.extra = {**(run.extra or {}), "error_type": "timeout"}
        raise
    except Exception as e:
        run.extra = {**(run.extra or {}), "error_type": "unknown", "error_message": str(e)}
        raise

Once errors are classified as metadata, you can build separate alert rules per error type in LangSmith, each with its own threshold and its own destination channel.

Latency: why p99 matters more than your average

Average latency is close to useless for user-facing LLM features, and if that's the only latency number in your dashboard, replace it. Here's why percentiles matter so much more here than in typical web services.

LLM latency distributions are heavily right-skewed. Most requests come back in a reasonable window, but a meaningful tail gets stuck behind provider-side queueing, long context processing, or a model deciding to write a much longer response than usual. Your average can look perfectly healthy — say, 2.5 seconds — while 5% of your users are sitting through 20-second waits. Those users are the ones who bounce, complain, or file support tickets, and averages hide them completely.

Track at minimum:

• p50 — the typical experience. Useful for spotting a systemic slowdown (e.g., you switched to a bigger model and everyone's baseline went up).
• p95 — your "normal bad day" tail. This is usually the number to alert on for user-experience regressions, because it reflects a meaningful chunk of real sessions without being so extreme it's dominated by one-off flukes.
• p99 — your worst-case tail. This is what tells you if there's a structural problem — a downstream retry storm, a provider having a bad day, a specific prompt template that occasionally explodes in token count.

Break these down by operation type, not just globally. A RAG pipeline's retrieval step, embedding call, and generation call have completely different latency profiles, and blending them into one "end-to-end latency" metric buries which stage actually regressed. LangSmith's trace trees make this straightforward — each span in the trace gets its own duration, so you can alert on "the retrieval step's p95 doubled" independently from "the generation step's p95 doubled."

A practical threshold-setting approach: don't pick round numbers out of the air. Pull two to four weeks of p95/p99 history, take that as your baseline, and set the alert at something like 1.5x baseline for a warning and 2.5x baseline for a page. Round numbers ("alert if p95 > 5s") work fine as a starting point but should get replaced with baseline-relative thresholds once you have real traffic data, because "acceptable latency" varies wildly depending on whether you're doing a simple classification call or a multi-step agent with tool use.

Cost per request: the metric that silently doubles

Cost is the metric most teams check monthly, via the invoice, which is exactly backwards — by the time the invoice arrives, whatever caused the spike has been running for weeks. Track cost per request (or cost per session, if that's the more meaningful unit for your product) as a live metric, trending over time, broken down by:

• Model — if you have automatic fallback to a more expensive model on failures, or a routing layer that picks models dynamically, cost per request should be tracked per model so a silent shift toward the expensive one doesn't hide inside a blended average.
• Endpoint or feature — a summarization feature and an open-ended chat feature have very different natural cost profiles; alert on each independently.
• Input vs. output tokens — a jump in output token cost with flat input tokens usually means the model started writing longer responses (prompt drift, a system prompt change, or a model version update from the provider). A jump in input token cost usually means someone's stuffing more context in — a RAG retrieval step returning more chunks than it should, or a conversation history that isn't being truncated.

LangSmith tracks token counts and, if you configure per-model pricing, computed cost per run natively, so this can ride on the same trace data you're already collecting for latency and errors — no separate billing pipeline needed. Set the alert as a percentage change against a rolling baseline (e.g., "average cost per request up more than 30% over the trailing 24 hours compared to the trailing 7 days") rather than an absolute dollar figure, since absolute cost naturally grows with traffic and a static threshold will either be constantly noisy or so loose it never fires.

Quality drift: monitoring what the model actually says, continuously

This is the layer that separates real production monitoring from infrastructure monitoring wearing an LLM costume, and it's the one most teams skip because it's the hardest to automate. The trap is thinking that evaluation is something you do once, at deploy time, against a curated test set, and then you're done. That test set tells you the model was good on the day you checked. It tells you nothing about whether the model is still good three weeks later, after:

• The upstream model provider silently updated the underlying model behind a version alias.
• Your RAG index picked up new, lower-quality documents.
• Real user inputs started drifting away from the shapes your test set covered.
• A prompt template got edited by someone who didn't run the eval suite.

The fix is continuous evaluation on a sample of live production traffic, not just a one-time gate. The idea: pull a random (or stratified) sample of production traces on a schedule, run them through an automated evaluator — often another LLM acting as a judge, sometimes a simpler rules-based check — and track the resulting quality score as a time series, exactly like you'd track latency or error rate. A drop in that score is a quality regression alert, and it's often the earliest signal you get of a real problem, well before it shows up as a support ticket spike.

Here's a conceptual version of what that sampling job looks like:

import random
from datetime import datetime, timedelta
from langsmith import Client

client = Client()

def sample_and_evaluate(hours_back: int = 1, sample_rate: float = 0.05):
    """Pull recent production runs, sample a slice, score them, and log results."""
    since = datetime.utcnow() - timedelta(hours=hours_back)

    runs = client.list_runs(
        project_name="production",
        start_time=since,
        run_type="chain",
        filter='eq(status, "success")',
    )

    sampled = [r for r in runs if random.random() < sample_rate]

    results = []
    for run in sampled:
        score = judge_output(
            input_data=run.inputs,
            output_data=run.outputs,
        )
        results.append({
            "run_id": run.id,
            "score": score.value,
            "reasoning": score.reasoning,
            "timestamp": run.start_time,
        })
        client.create_feedback(
            run_id=run.id,
            key="continuous_quality_judge",
            score=score.value,
            comment=score.reasoning,
        )

    avg_score = sum(r["score"] for r in results) / max(len(results), 1)
    if avg_score < QUALITY_THRESHOLD:
        fire_alert(
            severity="warning",
            message=f"Quality score dropped to {avg_score:.2f} over last {hours_back}h "
                     f"({len(results)} samples)",
        )
    return results


def judge_output(input_data: dict, output_data: dict):
    """LLM-as-a-judge: scores a single production output against a rubric."""
    judge_prompt = f"""
    Evaluate this AI response on a scale of 0-1 for helpfulness, accuracy,
    and adherence to instructions. Input: {input_data}. Output: {output_data}.
    Return a score and a one-sentence reason.
    """
    # call your judge model here, parse structured score + reasoning
    ...

Run this on a schedule — every hour is a reasonable starting cadence for most products, tightening to every 15 minutes for high-traffic or high-risk features. The sample rate can be small; you're building a trend line, not auditing every request. Feed the resulting scores into LangSmith as feedback on the original runs, so a human reviewing a specific trace later can see both what happened and how it scored, in one place.

The key discipline here: define your rubric once, keep it stable, and version it when you do change it. A quality score is only useful as a trend if the yardstick isn't moving under you. If you rewrite your judge prompt, treat that like a metric definition change — note when it happened, expect a step change in the score around that date, and don't read the step change as a real quality shift.

Token usage anomalies: your canary for prompt injection and runaway loops

Token usage deserves its own alert category, separate from cost, because an anomaly here is often a security or correctness signal, not just a billing signal.

Watch for:

• Sudden spikes in input tokens per request — a common signature of prompt injection attempts, where an attacker is trying to stuff long adversarial instructions into a field that's supposed to hold a short user query. It also shows up with retrieval bugs, where a RAG step starts returning far more chunks than intended.
• Sudden spikes in output tokens per request — the classic sign of a runaway loop in an agent: the model gets stuck repeating a tool call, or a self-correction loop never terminates and keeps generating until it hits your max-token cap. If you see requests consistently landing exactly at your configured max output tokens, that's not a coincidence — that's the model being cut off mid-loop, and you're paying full price for it every time.
• Anomalous token counts for a specific user or API key — useful for catching abuse, scraping, or a single integration partner sending malformed requests at scale.

Set this up as a statistical anomaly alert rather than a fixed threshold where possible — z-score against a rolling window per endpoint, since "normal" token counts vary enormously by feature. A summarizer with a 200-word cap and a long-form agent have nothing in common on this axis, so a single global threshold will either miss real anomalies in the low-volume feature or spam you constantly from the naturally high-variance one.

Setting alert thresholds that people actually trust

An alert that fires constantly and turns out to be nothing trains everyone to ignore it, and that's worse than having no alert at all — it means the one time it matters, someone will dismiss it anyway. A few rules that hold up in practice:

• Baseline against your own history, not absolute numbers. Pull 2-4 weeks of data for whatever metric you're alerting on, and set thresholds relative to that baseline (percentage change, standard deviations, etc.) rather than a number someone guessed in a meeting.
• Require sustained breach, not a single data point. A latency spike lasting 30 seconds during a traffic burst is not the same problem as p95 staying elevated for 20 minutes. Alert on the latter; log the former.
• Separate "degraded" from "broken." A 5% error rate increase and a 50% error rate increase are different emergencies calling for different responses. Build two thresholds per metric where it makes sense — a warning tier and a critical tier — rather than one binary alarm.
• Revisit thresholds monthly. Traffic patterns, model versions, and product usage all shift. A threshold that made sense at launch will eventually be either constantly tripping or completely dead. Put a recurring calendar reminder on this; it's the kind of maintenance that never happens unless it's scheduled.

Routing: not everything deserves a 3am page

The fastest way to burn out an on-call rotation is treating every alert as equally urgent. Build explicit severity tiers and route accordingly:

• Page (wakes someone up) — reserved for things actively breaking the product for real users right now: API error rate above a critical threshold, p99 latency indicating widespread timeouts, a full provider outage. This tier should be rare. If it's paging more than once every couple of weeks in steady state, your thresholds are too tight or your infra is genuinely unstable — figure out which.
• Urgent Slack/Teams alert (business hours response expected) — cost anomalies, moderate error rate increases, quality score drops that are clearly trending but not catastrophic. Someone should look today, not necessarily right now.
• Dashboard/digest (reviewed on a cadence) — slow drift metrics, weekly quality trend summaries, token usage patterns worth knowing about but not urgent. A daily or weekly digest works fine here.

In LangSmith, this maps to configuring separate alert rules per metric with different destinations — PagerDuty or an on-call webhook for the page tier, a dedicated Slack channel for the urgent tier, and a scheduled report or a lower-traffic channel for the digest tier. Resist the temptation to route everything into one #alerts channel "for visibility" — that channel becomes background noise within a week, and the actual pages get lost in it. If the volume in a channel means nobody's really reading it, that channel has failed at its one job.

Name the channels by response expectation, not by system: #llm-page (drop everything), #llm-urgent (today), #llm-digest (whenever). This makes the expected response obvious to anyone new who joins the rotation, without them needing tribal knowledge about which channel means what.

The runbook: what to actually do when the alert fires

An alert without a runbook just tells someone to feel anxious. For every alert you configure, write down — before you need it, not while you're paging through logs at 2am — the answers to:

1. What does this alert mean, concretely? Not "error rate high" but "the classify-intent step is failing to parse JSON output more than 10% of the time over 15 minutes."
2. What's the first thing to check? Usually: is this isolated to one model/endpoint/version, or is it global? LangSmith's trace filtering by model, tag, and time window makes this a two-minute check if you know to do it — link the exact saved filter in the runbook so nobody has to reconstruct it under pressure.
3. What's the likely cause, ranked by probability? For a quality score drop: recent prompt change (check deploy history first), upstream model version change (check provider status page), or a genuine shift in user input patterns (pull recent low-scoring traces and read them).
4. What's the safe mitigation while you investigate? Often this is "roll back the last prompt/config change" or "fail over to the previous model version" — decide this in advance, not during the incident, and make sure the rollback mechanism actually exists and has been tested.
5. Who else needs to know, and when? A quality regression affecting a customer-facing feature might need a heads-up to support before tickets start rolling in, not after.
6. How do we confirm it's resolved? Tie this back to the same metric that fired the alert — don't declare victory until the dashboard agrees with you.

Keep runbooks next to the alert definitions themselves, not in a separate wiki that goes stale. If you're using LangSmith's alerting, link the runbook directly in the alert's description field so whoever's on call gets it in the same notification that woke them up, not a search away.

Bringing it together: a monitoring setup that earns trust

None of these layers work in isolation. Error rates tell you something's broken. Latency percentiles tell you the experience is degrading even when nothing's technically "down." Cost and token anomalies tell you something structural shifted, sometimes for security reasons. And continuous quality evaluation is the layer that catches the failures with no error, no latency spike, and no cost change at all — just a model that's quietly gotten worse at its actual job.

The teams that get burned by production LLM issues are almost always the ones that shipped strong infrastructure monitoring and stopped there, treating the deploy-time eval suite as a one-time gate rather than an ongoing process. The fix isn't exotic. It's disciplined: classify your errors, watch percentiles instead of averages, trend your cost and token usage against a real baseline, and run LLM-as-a-Judge evaluation against a live sample of production traffic on a schedule, not just before you ship. Wire the results into alert tiers that route to the right channel with the right urgency, and back every alert with a runbook that tells whoever's on call exactly what to do next. That's what turns LangSmith from a place you look at traces after something goes wrong into a system that tells you before your customers do.