LangSmith Webhooks and Alerts: Automating Incident Response

Your LLM app failed at 2 a.m. and nobody noticed until a customer complained the next morning. The traces were all there in LangSmith the whole time — error rates spiking, latency climbing, feedback scores tanking — but nobody was looking at a dashboard in the middle of the night. This is the gap between observability and operations: collecting traces tells you what happened, but only alerting tells you *when to act*. LangSmith webhooks and alerts close that gap. In this guide, we will configure threshold-based alerts on error rate, latency, and feedback scores, wire them to webhooks and PagerDuty, build a webhook receiver that triages incidents automatically, and use automation rules to capture the evidence you need to actually fix the problem — not just know about it.

Why LLM Applications Need a Different Kind of Alerting

Traditional alerting was built for deterministic systems. A web server returns 500s, a database connection pool exhausts, a disk fills up — these are binary, unambiguous failure states, and decades of tooling exist to page someone when they occur.

LLM applications fail differently, and often silently. Consider the failure modes you actually see in production:

• Hard errors — the model provider returns a 429 rate limit or a 529 overloaded error, a tool call throws an exception, a chain times out. These look like traditional failures and are the easiest to alert on.
• Latency degradation — the provider is up but slow, or your retrieval step starts scanning a bloated vector index, and P99 generation time quietly triples. Users do not file bug reports for slow; they just leave.
• Quality regressions — the model responds quickly and without errors, but the answers got worse. A silent provider-side model update, a prompt change that regressed an edge case, or a retriever returning stale documents. Nothing in your infrastructure metrics moves at all.
• Feedback collapse — thumbs-down rates creep up over hours or days. By the time someone eyeballs a dashboard, thousands of bad interactions have already happened.

The first two categories map reasonably well to classic APM. The last two do not — no CPU graph in the world tells you your RAG pipeline started hallucinating. This is exactly where LangSmith sits: because it already captures every run with its error status, latency, token usage, and attached feedback scores, it can evaluate alert conditions against the *semantics* of your LLM application, not just its infrastructure. Alerts on error rate catch the hard failures; alerts on feedback score and latency catch the soft ones that traditional monitoring is structurally blind to.

The goal of this article is a pipeline that looks like this: LangSmith detects a threshold breach, fires a webhook, your receiver enriches and routes the incident, the on-call engineer gets paged with links to the exact failing traces, and an automation rule has already siphoned those failing runs into an annotation queue for the postmortem.

How LangSmith Alerts Work

LangSmith alerts are configured per tracing project, which is a sensible scope: your production chatbot, your internal summarization service, and your staging environment are different projects with different tolerance for failure.

Each alert is built from a few core pieces:

• A metric. LangSmith supports alerting on error count/rate (runs whose status is error), latency (how long runs take to complete), and feedback scores (values attached to runs by users or by online evaluators).
• Filters. You can scope the alert to a subset of runs — for example, only root runs, only runs with a particular name or tag, so a flaky experimental chain does not page anyone while your checkout-assistant chain is held to a strict standard.
• An aggregation window. Metrics are evaluated over a rolling window of recent runs rather than on individual events. This matters enormously for LLM workloads: a single timeout is noise, fifteen timeouts in five minutes is an incident.
• A threshold and condition. Error rate above X percent, average latency above N seconds, average feedback score below some floor.
• A notification channel. This is where webhooks and PagerDuty come in — the alert needs somewhere to go.

The window-based evaluation is the piece newcomers underestimate. LLM providers throw transient errors constantly; retries absorb most of them. If you alert on every single error event, you will train your team to ignore the channel within a week. Aggregation windows let you encode "this is now a pattern, not a blip" directly into the alert definition.

A practical starter set for a production project looks like this:

1. Error rate alert — error percentage over a short window exceeds your normal baseline. This is your smoke detector, and it should page.
2. Latency alert — average or high-percentile run latency exceeds a threshold over the window. This usually starts as a Slack notification and gets promoted to paging once you trust the threshold.
3. Feedback score alert — the average feedback score across recent runs drops below a floor. This is your quality regression detector, and it is the alert traditional tooling cannot give you.

Set thresholds empirically, not aspirationally. Look at your project's actual monitoring charts over the past couple of weeks, find the worst normal-operations value, and set the threshold meaningfully beyond it. An alert that fires during normal operation is worse than no alert.

Webhooks: The Building Block for Automation

LangSmith offers PagerDuty integration for direct paging, but the webhook channel is where automation lives, because a webhook can trigger *anything*.

When an alert fires, LangSmith sends an HTTP POST to a URL you configure. The request carries a JSON payload describing the alert — which alert fired, which project it belongs to, the metric and threshold involved, and when it triggered. You provide the URL, and you can typically configure custom headers, which is how you attach a shared secret so your receiver can reject forged requests.

That single POST is a programmable hook into your entire operational stack. Teams commonly use it to:

• Post a formatted message into a Slack or Microsooft Teams incident channel with a direct link to the LangSmith project.
• Open a ticket in Jira or Linear automatically, pre-populated with the alert context.
• Trigger a runbook: flip a feature flag to fall back to a more reliable model, disable an experimental prompt variant, or scale up a rate-limited resource.
• Feed an internal incident-management service that handles deduplication, escalation, and status pages.

The mental model to adopt: the alert is the trigger, the webhook receiver is the brain. LangSmith decides *when* something is wrong; your receiver decides *what to do about it*. Keeping that logic in your own service — rather than trying to encode it in alert configuration — means you can evolve your response playbook without touching the alerts themselves.

One operational note: your receiver should acknowledge the webhook fast. Do the minimal work needed to accept and persist the event, return a 2xx, and do the heavy lifting (Slack calls, ticket creation, trace fetching) asynchronously. A webhook endpoint that takes thirty seconds to respond because it is calling four downstream APIs inline is a webhook endpoint that will eventually drop events.

Building a Webhook Receiver That Triages Incidents

Let us build a real receiver. The design goals: verify the request is genuinely from our LangSmith alert configuration, classify the severity, route hard failures to PagerDuty and soft failures to Slack, and deduplicate so a flapping alert does not create ten incidents.

import hmac
import os
import time
import httpx
from fastapi import BackgroundTasks, FastAPI, Header, HTTPException, Request

app = FastAPI()

WEBHOOK_SECRET = os.environ["ALERT_WEBHOOK_SECRET"]
SLACK_WEBHOOK_URL = os.environ["SLACK_WEBHOOK_URL"]
PAGERDUTY_ROUTING_KEY = os.environ.get("PAGERDUTY_ROUTING_KEY", "")

# alert_id -> last handled timestamp, for dedup
_recently_handled: dict[str, float] = {}
DEDUP_WINDOW_SECONDS = 900  # suppress repeats for 15 minutes


def classify(alert: dict) -> str:
    """Map an alert payload to a severity level."""
    metric = (alert.get("metric") or "").lower()
    if "error" in metric:
        return "page"          # hard failures wake someone up
    if "latency" in metric:
        return "notify"        # degradation goes to Slack
    if "feedback" in metric or "score" in metric:
        return "notify"        # quality regressions go to Slack
    return "notify"


async def notify_slack(alert: dict) -> None:
    text = (
        f":rotating_light: *LangSmith alert: {alert.get('name', 'unnamed')}*\n"
        f"Project: `{alert.get('project', 'unknown')}`\n"
        f"Metric: `{alert.get('metric')}` breached threshold "
        f"`{alert.get('threshold')}` (observed `{alert.get('value')}`)\n"
        f"Triage: open the tracing project, filter to errored runs "
        f"in the alert window, and check the annotation queue."
    )
    async with httpx.AsyncClient(timeout=10) as client:
        await client.post(SLACK_WEBHOOK_URL, json={"text": text})


async def page_oncall(alert: dict) -> None:
    if not PAGERDUTY_ROUTING_KEY:
        await notify_slack(alert)  # fail open to Slack
        return
    event = {
        "routing_key": PAGERDUTY_ROUTING_KEY,
        "event_action": "trigger",
        "dedup_key": f"langsmith-{alert.get('id', alert.get('name'))}",
        "payload": {
            "summary": f"LangSmith: {alert.get('name')} breached "
                       f"({alert.get('metric')})",
            "source": alert.get("project", "langsmith"),
            "severity": "error",
            "custom_details": alert,
        },
    }
    async with httpx.AsyncClient(timeout=10) as client:
        await client.post(
            "https://events.pagerduty.com/v2/enqueue", json=event
        )


@app.post("/hooks/langsmith-alert")
async def handle_alert(
    request: Request,
    background: BackgroundTasks,
    x_alert_token: str = Header(default=""),
):
    # 1. Authenticate: reject anything without our shared secret header
    if not hmac.compare_digest(x_alert_token, WEBHOOK_SECRET):
        raise HTTPException(status_code=401, detail="bad token")

    alert = await request.json()

    # 2. Deduplicate flapping alerts
    key = str(alert.get("id") or alert.get("name") or "unknown")
    now = time.time()
    last = _recently_handled.get(key, 0)
    if now - last < DEDUP_WINDOW_SECONDS:
        return {"status": "suppressed_duplicate"}
    _recently_handled[key] = now

    # 3. Route asynchronously and ACK immediately
    if classify(alert) == "page":
        background.add_task(page_oncall, alert)
    else:
        background.add_task(notify_slack, alert)
    return {"status": "accepted"}

Deploy this behind HTTPS, set the secret as a custom header in the LangSmith alert's webhook configuration, and point the alert at /hooks/langsmith-alert. A few implementation notes worth internalizing:

• The token check uses `hmac.compare_digest`, not ==, to avoid timing side channels. Anything reachable from the public internet needs authentication, and a webhook endpoint that pages your on-call is a denial-of-sleep vector if left open.
• Field names are read defensively with .get(). Treat any webhook payload as a contract you verify empirically: fire a test alert, log the raw body, and adjust your parsing to what actually arrives rather than what you assumed.
• Deduplication lives in your receiver. An alert hovering around its threshold can trigger repeatedly; the in-memory window above suppresses repeats (use Redis instead if you run multiple replicas). The PagerDuty dedup_key adds a second layer so repeated triggers update one incident instead of spawning many.
• The endpoint does nothing slow inline. Authenticate, dedupe, enqueue, return. Everything with a network call happens in a background task.

Test it before trusting it. Simulate the delivery with curl and confirm both the happy path and the auth rejection:

curl -X POST https://ops.example.com/hooks/langsmith-alert \
  -H "Content-Type: application/json" \
  -H "X-Alert-Token: $ALERT_WEBHOOK_SECRET" \
  -d '{"id": "alrt-1", "name": "prod error rate", "project": "checkout-assistant", "metric": "error_rate", "threshold": 0.05, "value": 0.11}'

Then trigger a real one: temporarily lower an alert threshold below the current metric value in a staging project, watch the webhook arrive, and restore the threshold. Never let a paging path go to production untested.

Routing Severity: Slack, PagerDuty, and the Escalation Ladder

With the receiver in place, the interesting decisions are policy, not code. A routing scheme that works well for LLM products:

• Page (PagerDuty, phone-buzzing severity): error-rate breaches on production, user-facing projects. When the model provider is down or your chain is throwing on every request, minutes matter — users are staring at spinners right now.
• Notify (Slack channel, business-hours attention): latency threshold breaches and feedback-score drops. These are real incidents, but the marginal value of waking someone at 3 a.m. for a quality regression is usually negative — the fix is a prompt change or a model rollback that needs a clear-headed human anyway.
• Log only (ticket or dashboard annotation): staging-project alerts and experimental chains. You want the record, not the interruption.

For teams already on PagerDuty, use LangSmith's native PagerDuty channel for the paging tier and reserve the webhook channel for everything programmable. For teams without formal on-call, the webhook-to-Slack path in the code above is a complete, honest starting point — just make sure the channel it posts to is one people treat as actionable, not one more muted firehose.

One pattern worth stealing from mature SRE practice: encode the *first triage step* directly in the notification text, as the Slack message above does. The engineer who gets paged should never have to wonder what to look at first. "Open the project, filter to errored runs in the window, check whether errors are concentrated in one chain or spread across all of them" is a runbook sentence that saves ten confused minutes per incident.

Automation Rules: Capture the Evidence While It Is Fresh

Alerts tell you something broke. Fixing it requires examples, and this is where LangSmith's automation rules complete the story.

An automation rule watches a tracing project for runs matching a filter — errored runs, runs with negative feedback, runs matching a tag, optionally sampled at some rate — and applies an action automatically. The actions are the valuable part:

• Add to an annotation queue, so a human reviews the failing interactions with full trace context and records structured judgments.
• Add to a dataset, turning production failures directly into regression test cases.
• Trigger a webhook, giving you run-level automation in addition to threshold-level alerting.
• Run an online evaluator, scoring live traffic with an LLM-as-judge so quality gets a numeric signal.

Pair each alert with a rule that gathers its evidence. If your error-rate alert fires at 2 a.m. and the on-call engineer mitigates by rolling back to the previous prompt, the postmortem still needs the failing traces. A rule that automatically routed every errored run into an annotation queue means the evidence was collected while the incident was live — nobody has to reconstruct it from memory, and nobody has to scroll through thousands of healthy traces to find the bad ones.

The dataset action closes the loop even more tightly. Every production failure that gets added to a dataset becomes a permanent regression case: before the next prompt or model change ships, you run your offline evaluation against that dataset and verify the old failure stays fixed. Incidents stop being pure cost and start compounding into a better test suite.

Note the two distinct webhook layers, because conflating them causes design mistakes: alert webhooks fire when an aggregate threshold is breached (one event per incident — right for paging and tickets), while automation rule webhooks fire per matching run (many events — right for feeding queues, datasets, or per-example pipelines). Page from the first, collect from the second.

Alerting on Quality, Not Just Errors

The most underused alert type is the feedback score alert, and it deserves its own treatment because it is the one that catches failures nothing else can see.

Feedback scores in LangSmith come from two sources. Human feedback — thumbs up/down buttons, star ratings, "was this helpful?" prompts — logged from your application against the run ID. Online evaluators — LLM-as-judge evaluators that LangSmith runs automatically against sampled live traffic, scoring dimensions like correctness, groundedness against retrieved context, or answer relevance, and attaching the result as a feedback score on the run.

Once scores are flowing, a feedback alert is just another threshold: if the average score across the recent window drops below your floor, fire the webhook. But consider what this actually gives you. A provider silently updates the model behind your API endpoint and its behavior shifts on your task. No error occurs. Latency is fine. Every infrastructure dashboard is green. The only signal in the universe that something changed is that your judge scores and thumbs-up rates moved — and your feedback alert catches it within one aggregation window.

Two pieces of practical advice for making quality alerts trustworthy:

• Calibrate the judge before alerting on it. An online evaluator with poorly written criteria produces noisy scores, and noisy scores produce false alarms. Validate the evaluator against a hand-labeled sample first; only wire an alert to a score you have seen agree with human judgment.
• Mind the volume. Feedback signals are sparser than error signals — not every run gets human feedback, and online evaluators typically sample rather than score everything. Use longer aggregation windows for feedback alerts than for error alerts, so a handful of grumpy users during a quiet hour does not constitute a statistical event.

Avoiding Alert Fatigue

Every alerting system dies the same death: too many notifications, then muted channels, then a real incident sails through unseen. LLM systems are especially prone to this because they are inherently noisy — transient provider errors, natural latency variance across prompt lengths, feedback that swings with traffic mix. Some discipline that keeps the system trustworthy:

• Every alert must be actionable. Before creating one, finish the sentence "when this fires, the responder will immediately…". If there is no answer, you are building a statistic, not an alert — put it on a dashboard instead.
• Thresholds come from observed baselines. Review your project's monitoring charts, find normal, and alert beyond it. Revisit quarterly; baselines drift as traffic and prompts evolve.
• Windows absorb noise. A five-minute error window catches real outages fast. A several-hour feedback window catches real regressions without twitching at variance.
• Separate the channels by consequence. One channel that pages, one channel that informs. The moment paging-severity and FYI-severity share a channel, humans recalibrate to the lower severity.
• Deduplicate aggressively. Flapping alerts — hovering at the threshold, firing every window — must collapse into a single incident, as the receiver code above does.
• Delete alerts that lie. Track which alerts fired last month and whether anyone acted. An alert with three false positives and zero true positives is a trust-destroyer; fix its threshold or remove it.

A small set of honest alerts beats a large set of noisy ones by a wide margin. Three alerts everyone trusts will catch more real incidents than thirty that everyone ignores.

From Alert to Fix: A Complete Incident Walkthrough

Let us trace the whole pipeline through a realistic incident to see how the pieces cooperate.

1. Detection. Tuesday, 21:40. Your production RAG assistant's error-rate alert breaches its window threshold — the embedding service behind your retriever started refusing connections after a botched deploy, and every retrieval-dependent run is erroring.
2. Delivery. LangSmith fires the alert webhook. Your receiver authenticates the request, sees an error-metric alert, classifies it as paging severity, and sends a PagerDuty event with a dedup key. Subsequent triggers during the incident update the same PagerDuty incident instead of stacking new ones.
3. Triage. The on-call engineer opens the tracing project from the notification and filters to errored runs in the alert window. The traces make the shape of the failure obvious in seconds: every failed run dies at the retriever step with the same connection error, while runs that skip retrieval succeed. This is the payoff of alerting *inside* your observability tool — the alert and the evidence live in the same place.
4. Mitigation. The engineer rolls back the embedding service deploy. Error rate falls back under the threshold within a few windows. The Slack channel gets a resolution note.
5. Evidence collection — already done. The automation rule targeting errored runs was quietly adding every failure to the incident annotation queue throughout. The postmortem has its exhibits without anyone lifting a finger at 21:40.
6. Regression-proofing. A representative slice of the failing interactions goes into a dataset. The team adds an offline evaluation step that exercises retrieval failure handling — the app should degrade gracefully when the retriever is down, not hard-error — and that gap becomes a fix with a permanent test.

Compare this against the counterfactual: no alerts, the failure runs until morning, someone reconstructs the timeline from memory, and the traces have long since scrolled into the noise. The difference is not the observability data — LangSmith captured identical traces in both worlds. The difference is the automation wrapped around it. Detection, routing, evidence collection, and regression-proofing all happened mechanically, which is precisely what "automating incident response" means: humans make the judgment calls, machines do everything else.

Start small and let the system earn trust. One error-rate alert with a webhook to Slack this week. PagerDuty routing and an evidence-collection rule next week. A calibrated online evaluator feeding a feedback alert the week after. Each layer builds on the previous one, and within a month your LLM application has an operational safety net most teams never get around to building.

If you want to go deeper — tracing fundamentals, online and offline evaluation, annotation queues, datasets, prompt experiments, and the full alerting and automation workflow covered in this article with hands-on projects — check out the LangSmith Tutorial course on teachyou.ai, where we build a production-grade LLM observability and incident-response pipeline from scratch, step by step.