LangSmith Feedback Collection: Capturing User Ratings on Responses

Your LLM app is in production. Latency dashboards look great, error rates are near zero, token spend is under control — and yet you have no idea whether users actually like the answers your chatbot gives them. Traditional observability tells you that a response was generated, not whether it was any good. The people best positioned to judge quality are your users, and every thumbs up, star rating, or angry "this is wrong" comment they leave is a labeled data point you are probably throwing away. LangSmith feedback collection fixes that. It lets you attach structured ratings, scores, comments, and even corrections directly to the trace that produced a response, so quality signals live right next to the prompts, retrieved documents, and model calls that caused them. In this guide, we will build user feedback capture from scratch: wiring up run IDs, calling create_feedback from the Python and TypeScript SDKs, collecting ratings safely from a frontend with pre-signed feedback tokens, and turning the accumulated signal into datasets, monitors, and better prompts.

What Feedback Means in LangSmith

In LangSmith, feedback is a first-class object attached to a run. A run is a single traced unit of work — an LLM call, a chain invocation, a full agent execution — and every run has a unique ID. When a user clicks thumbs down on a chatbot answer, you take the run ID of the trace that generated that answer and create a feedback record against it. From that moment, the rating is permanently linked to everything LangSmith knows about that execution: the exact prompt, the retrieved context, the model parameters, the intermediate tool calls, and the final output.

A feedback record has a small, deliberate schema. The most important fields are:

• key — the name of the feedback signal, such as user_rating, thumbs, correctness, or helpfulness. Keys let you record multiple independent signals on the same run.
• score — a numeric value. For binary thumbs feedback the convention is 1 for positive and 0 for negative. For star ratings you might use 1 through 5, or normalize to 0.0–1.0.
• value — a categorical value when a number does not fit, such as "hallucination" or "off_topic".
• comment — free-text explanation, typically the "tell us more" box users fill in after a thumbs down.
• correction — the answer the user believes should have been given. This field is gold for building fine-tuning and few-shot datasets later.

Feedback also carries a source type. Feedback created by end users or your application code is API-sourced; feedback added by reviewers in the LangSmith UI is app-sourced; feedback produced by automated evaluators (LLM-as-judge, heuristics) is model-sourced. All three land in the same place, which means human ratings and automated evaluation scores can be compared side by side on identical traces. That comparison is how teams validate whether their automated evaluators actually agree with real humans — and it is only possible because feedback collection funnels everything onto the run.

The mental model to keep: traces answer "what happened", feedback answers "was it good". You need both to run an LLM product seriously.

Why User Ratings Beat Offline Evals Alone

Offline evaluation — running a curated dataset through your chain and scoring outputs — is essential, but it has a blind spot: it only measures the questions you thought to include. Production users ask things you never anticipated, phrase them in ways your dataset does not cover, and care about qualities (tone, brevity, formatting) that your correctness evaluator ignores.

User feedback collection complements offline evals in several concrete ways:

• It samples the true input distribution. Every rating comes from a real query a real user cared enough to ask, so your quality signal is automatically weighted toward what matters in production.
• It catches regressions that evals miss. If a prompt change tanks answer tone, your correctness evals may stay green while thumbs-down rates climb within hours.
• It generates labeled data for free. Runs with negative feedback are candidate test cases; runs with corrections are candidate training examples. LangSmith lets you filter runs by feedback and add them to a dataset in a couple of clicks or one API call.
• It grounds prioritization. Instead of debating which failure mode to fix first, you sort by feedback volume and read the worst-rated traces.

There is one caveat worth internalizing before you build anything: explicit feedback is sparse and biased. Most users never click either thumb, and the ones who do skew toward the annoyed. This does not make the signal useless — it makes it directional. Treat feedback rates as trend indicators and triage queues, not as an unbiased estimate of overall satisfaction. Later in this article we will cover implicit feedback (copy events, retries, session abandonment) that partially compensates for the sparsity.

Setting Up: Environment, Tracing, and Run IDs

Feedback attaches to runs, so before you can collect a single rating you need tracing enabled and a reliable way to know which run produced which response. Install the SDK and set the standard environment variables:

pip install -U langsmith langchain langchain-openai
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="<your-langsmith-api-key>"
export LANGSMITH_PROJECT="support-bot-prod"

With tracing on, every chain invocation creates a trace in your project. The problem is plumbing: the run ID exists on the server, and your feedback endpoint — which will be called seconds or minutes later when the user clicks a thumb — needs that exact ID. There are three reliable patterns.

The first and most robust pattern is to generate the run ID yourself before invoking the chain. Both LangChain and the LangSmith SDK accept a client-supplied UUID, which means you never have to fish the ID back out of anything:

import uuid
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a concise support assistant."),
    ("human", "{question}"),
])
chain = prompt | ChatOpenAI(model="gpt-4o-mini")

run_id = str(uuid.uuid4())
answer = chain.invoke(
    {"question": "How do I rotate my API key?"},
    config={"run_id": run_id},
)
# Return both to the frontend; the run_id rides along with the answer.

The second pattern uses collect_runs as a context manager when you would rather let LangChain assign IDs:

from langchain_core.tracers.context import collect_runs

with collect_runs() as cb:
    answer = chain.invoke({"question": "How do I rotate my API key?"})
    run_id = cb.traced_runs[0].id

The third pattern applies when you trace plain Python functions with the @traceable decorator instead of LangChain. Inside a traced function you can grab the current run tree and read its ID:

from langsmith import traceable
from langsmith.run_helpers import get_current_run_tree

@traceable(name="answer_question")
def answer_question(question: str) -> dict:
    run_tree = get_current_run_tree()
    response = generate_answer(question)  # your model call
    return {"answer": response, "run_id": str(run_tree.id)}

Whichever pattern you choose, the contract with your frontend is the same: every response payload includes the run ID, and the frontend echoes it back when the user rates the response. Store it in component state next to the message; do not try to reconstruct it later from timestamps or content matching.

One practical tip for chat applications: attach feedback to the top-level trace (the root run) rather than a nested LLM call. Users rate the whole answer, and the root run gives reviewers the complete picture when they open the trace from a feedback filter.

Logging Ratings with create_feedback

With run IDs flowing to the frontend and back, recording a rating is a single SDK call. create_feedback is the workhorse of LangSmith feedback collection, and a minimal thumbs endpoint looks like this:

from fastapi import FastAPI
from pydantic import BaseModel
from langsmith import Client

app = FastAPI()
ls_client = Client()

class FeedbackPayload(BaseModel):
    run_id: str
    score: int            # 1 = thumbs up, 0 = thumbs down
    comment: str | None = None

@app.post("/api/feedback")
async def submit_feedback(payload: FeedbackPayload):
    ls_client.create_feedback(
        run_id=payload.run_id,
        key="user_rating",
        score=payload.score,
        comment=payload.comment,
        source_info={"channel": "web_chat", "app_version": "2.4.1"},
    )
    return {"ok": True}

A few details in that call deserve attention. The key is your naming contract — pick keys once, document them, and keep them stable, because every chart, filter, and automation you build later references them by exact string. The score convention of 1/0 for thumbs is what LangSmith's built-in aggregations expect for binary signals. The source_info dictionary is free-form metadata about where the feedback came from; it costs nothing to include and pays off the first time you need to compare satisfaction across your web app and Slack bot.

Richer signals use the other fields. A five-star rating with an optional correction:

ls_client.create_feedback(
    run_id=payload.run_id,
    key="star_rating",
    score=payload.stars / 5.0,       # normalize 1-5 stars to 0.2-1.0
    comment=payload.comment,
    correction={"expected_answer": payload.suggested_answer}
    if payload.suggested_answer
    else None,
)

Categorical feedback — for example a "what went wrong?" picker offered after a thumbs down — uses value instead of score:

ls_client.create_feedback(
    run_id=payload.run_id,
    key="failure_mode",
    value=payload.reason,   # "hallucination" | "outdated" | "off_topic" | "formatting"
)

The TypeScript SDK mirrors the same API for Node backends:

import { Client } from "langsmith";

const client = new Client();

await client.createFeedback(runId, "user_rating", {
  score: 1,
  comment: "Answered in one line, exactly what I needed",
  sourceInfo: { channel: "web_chat" },
});

Two operational notes. First, create_feedback accepts a run ID for a run that has not finished being ingested yet — feedback submission does not need to wait for trace ingestion to complete, so you can fire it immediately without race-condition worries. Second, keep the call out of your request hot path where possible: enqueue it, run it in a background task, or at minimum wrap it in a try/except. A LangSmith hiccup should never break your user's chat experience, and feedback delivery is tolerant of a few seconds of delay.

You can also update feedback after the fact with update_feedback (for example, when a user edits their rating) and delete it with delete_feedback, using the feedback ID returned by the create call. If you plan to support editable ratings, persist that feedback ID alongside the message in your own database.

Collecting Feedback from the Frontend with Pre-signed Tokens

The architecture above routes every rating through your backend, which works but has costs: you maintain an endpoint, you handle auth, and your LangSmith API key must stay server-side because shipping it to a browser would expose your entire workspace. LangSmith offers a neat alternative for exactly this situation: pre-signed feedback tokens.

The idea is simple. When your backend returns a response, it also mints a short-lived, single-purpose URL that permits one specific feedback key to be written to one specific run. The frontend can POST directly to that URL — no API key, no custom endpoint, no ability to do anything else in your workspace.

from langsmith import Client

ls_client = Client()

def mint_feedback_token(run_id: str) -> str:
    token = ls_client.create_presigned_feedback_token(
        run_id=run_id,
        feedback_key="user_rating",
    )
    return token.url   # hand this to the frontend with the answer

On the browser side, submitting the rating is a plain fetch with no credentials:

async function sendRating(feedbackUrl, isPositive, comment) {
  await fetch(feedbackUrl, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      score: isPositive ? 1 : 0,
      comment: comment || undefined,
    }),
  });
}

If you prefer to keep token redemption server-side but still want the convenience, the Python SDK can consume a token directly with create_feedback_from_token(token_or_url, score=..., comment=...).

Pre-signed tokens shine in a few scenarios. Static or serverless frontends that have no natural place for a feedback endpoint. Emails and notification messages where you want one-click "was this helpful?" links — mint two tokens per message, one wired to score 1 and one to score 0, and embed them as links. Third-party surfaces like Slack or Discord bots where adding interactive buttons is easy but adding authenticated backend routes is annoying.

The trade-offs are worth knowing. Tokens have expirations (you can set a custom one at mint time), each token is bound to a single feedback key, and minting one token per response adds a small amount of latency to your response path — negligible for chat, worth batching for high-throughput APIs. For most teams, the backend-endpoint pattern and the token pattern coexist: tokens for lightweight surfaces, the endpoint for the main app where you already have sessions and want to enrich feedback with your own user metadata.

Designing a Feedback UX Users Will Actually Use

The engineering is the easy half. Feedback collection lives or dies on interaction design, because response rates for LLM feedback widgets are notoriously low and every design mistake lowers them further.

Principles that consistently help:

• Make the primary action one click. Thumbs up/down directly under the response, always visible, no hover-to-reveal on mobile. Every additional step cuts completion drastically.
• Ask for elaboration after capturing the rating, never before. Record the thumbs down immediately via create_feedback, then expand an optional comment box. If the user walks away, you still have the score; if they type, call update_feedback or log a second key with the comment.
• Use structured follow-ups over free text where you can. A four-option "what went wrong?" chip row (wrong answer, outdated info, didn't understand me, too long) produces categorical value feedback you can chart, whereas free text needs an LLM pass before it aggregates.
• Time in-conversation prompts carefully. Asking after every message is noise; a good default is rating affordances on every message but an explicit nudge only at natural completion points, like after the user says thanks or closes the session.
• Close the loop visibly. A brief "thanks — this helps improve answers" acknowledgment measurably improves repeat feedback behavior. Users keep rating when they believe someone is listening.

Alongside explicit ratings, instrument implicit signals — they arrive at ten to a hundred times the volume. A user copying the response to their clipboard is a strong positive; log it as key="copied", score=1. A user immediately rephrasing the same question is a soft negative; regeneration clicks are a stronger one. Session abandonment right after an answer, clicking a cited source, accepting a suggested edit — all of these map cleanly onto feedback keys with scores, and because they are just create_feedback calls with different keys, the pipeline you built in the previous sections handles them without modification. Keep implicit and explicit signals under separate keys so you never conflate "users who bothered to click" with "users who silently voted with their behavior".

From Ratings to Improvements: Using Feedback Inside LangSmith

Collecting ratings is pointless if they sit unread. LangSmith gives you several ways to convert accumulated feedback into concrete product improvements.

Filtering and triage. In the project view, filter runs by feedback — for example, all runs where user_rating has score 0 in the last 7 days. This is your triage queue. Because feedback is attached to full traces, each item in the queue opens into the complete execution: you see the retrieval step that pulled the wrong document or the system prompt that invited the hallucination, not just the bad output.

Dashboards and monitoring. Feedback scores aggregate into time-series charts, so you can watch the average of user_rating by day and set alerting on it. This turns user satisfaction into an SLO-like signal: deploy a prompt change, watch the feedback line, roll back if it dips. Group by metadata (model version, prompt version, A/B arm recorded in source_info) to compare variants on real traffic.

Building datasets from feedback. The highest-leverage move: filter to badly rated runs and add them to a dataset. Those inputs become regression tests — the next time you iterate on your prompt, your offline eval suite includes the exact queries that failed in production. Runs where users supplied a correction are even better: the input plus the corrected output is a ready-made ground-truth example. You can automate this with the SDK:

from langsmith import Client

ls_client = Client()

runs = ls_client.list_runs(
    project_name="support-bot-prod",
    filter='and(eq(feedback_key, "user_rating"), eq(feedback_score, 0))',
    is_root=True,
)

dataset = ls_client.read_dataset(dataset_name="thumbs-down-regressions")
for run in runs:
    ls_client.create_example(
        inputs=run.inputs,
        outputs=run.outputs,
        dataset_id=dataset.id,
    )

Annotation queues. Feedback-flagged runs can feed annotation queues, where domain experts review traces and attach their own richer feedback in the UI. A common workflow: user thumbs-down sends the run to a queue, a support engineer reviews it, labels the failure mode, and writes the correct answer — which then flows into your golden dataset.

Calibrating automated evaluators. Once you have a few hundred human ratings, run your LLM-as-judge evaluator over the same traces and compare its scores with the human ones, key by key. Where they disagree, read the traces and fix the judge prompt. Human feedback is the ground truth that keeps automated evaluation honest.

Common Mistakes and How to Avoid Them

A handful of failure patterns show up repeatedly in teams adopting LangSmith feedback collection. Skim this list before you ship.

1. Losing the run ID. If the frontend cannot associate a message with its run ID, feedback becomes unattachable. Generate the UUID server-side before invocation, return it in the response payload, and store it with the message in client state and in your own database.
2. Inconsistent feedback keys. user_rating, user-rating, and userRating are three different signals to LangSmith. Define keys in one shared constants module and import them everywhere, including your analytics code.
3. Blocking the request path. A synchronous create_feedback call inside your chat handler couples user experience to LangSmith availability. Use background tasks or a queue, and swallow-and-log errors.
4. Mixing scales under one key. If quality sometimes means a 0/1 thumb and sometimes a 1–5 star, its average is meaningless. One key, one scale, forever — normalize stars to 0–1 at write time if you want cross-signal comparability.
5. Rating the wrong run. Attaching feedback to a nested LLM call instead of the root run fragments your data. Users judge the final answer; rate the trace root.
6. Treating feedback rates as satisfaction. A 4 percent thumbs rate dominated by annoyed users is a triage signal, not a CSAT survey. Report trends and volumes, not "96 percent of users are happy".
7. Collecting and never reading. Decide upfront who reviews the thumbs-down queue and on what cadence, and wire the dataset-building automation early. Feedback that no one acts on trains users to stop giving it.
8. Ignoring abuse and noise. Public-facing feedback endpoints get spam. Rate-limit per session, validate that the run ID belongs to the requesting user's session, and prefer pre-signed tokens for anonymous surfaces since each token is single-purpose by construction.
9. Forgetting privacy. Comments and corrections are user-generated content and may contain personal data. Apply the same PII policies to feedback fields that you apply to traces, and scrub before logging if your compliance posture requires it.

None of these are hard to avoid individually; the trick is knowing they exist before the data is corrupted, because feedback history — unlike code — cannot be refactored after the fact.

Wrapping Up

Feedback collection is the shortest path from "our LLM app works" to "our LLM app is getting better every week". The mechanics are small — a run ID returned with each response, a create_feedback call when the user reacts, pre-signed tokens where a backend endpoint is inconvenient — but the compounding effect is large: a triage queue of real failures, dashboards that catch quality regressions within hours, and datasets built from the exact queries your users cared about. Start with a single binary key on your highest-traffic surface, keep the schema disciplined, and add structured follow-ups and implicit signals once the basic loop is running.

If you want to go deeper — tracing internals, evaluator design, annotation queues, dataset versioning, and production monitoring built on top of the feedback pipeline you just learned — the LangSmith Tutorial course on teachyou.ai walks through all of it with hands-on projects, including a complete feedback-instrumented chat application you can adapt for your own product. Capturing what your users think is step one; the course shows you how to turn that signal into a continuously improving system.