LangSmith Annotation Queues: Human Review at Scale

Every LLM team eventually hits the same wall. You ship an agent or a RAG pipeline, traces pile up in LangSmith by the thousands, and someone on the team says "we should really look at these." So an engineer opens the tracing project, scrolls through a few dozen runs, eyeballs the outputs, and forms a vibe-based opinion about quality. Two weeks later a different engineer does the same thing, reaches a different conclusion, and nobody wrote anything down either time. That is not evaluation. That is tourism. LangSmith annotation queues exist to fix exactly this: they turn trace review from a random walk into a structured, assignable, measurable workflow where humans label real production runs against a defined rubric, and every label lands somewhere useful — a feedback score, a dataset example, or ground truth for aligning an LLM-as-judge. In this guide we will build that workflow end to end: creating queues, feeding them automatically, designing rubrics reviewers can actually follow, and closing the loop so human judgment compounds instead of evaporating.

Why Human Review Still Matters in an LLM Pipeline

It is tempting to believe you can automate quality assessment entirely. Wire up an LLM-as-judge, score every trace, put the number on a dashboard, done. In practice, automated evaluation without a human anchor drifts into fiction. An LLM judge is just another prompt, and like any prompt it has blind spots: it may reward confident-sounding hallucinations, penalize correct answers phrased tersely, or miss domain-specific failures that only someone who understands your product would catch. The judge needs calibration, and calibration requires human labels to calibrate against.

Human review is also the only reliable way to discover failure modes you did not anticipate. Offline eval datasets test the failures you already know about. Production traces contain the failures you do not. A support bot that suddenly starts recommending a competitor, a SQL agent that quietly returns an empty result set instead of erroring, a summarizer that inverts the sentiment of a customer complaint — these are the kinds of problems that surface when a person actually reads transcripts, and almost never when a metric aggregates them away.

The problem has never been whether humans should review LLM outputs. The problem is logistics. Who reviews what? How do you avoid two people reviewing the same run while a thousand others go unread? What questions should reviewers answer, and where do their answers go? Without tooling, human review collapses under its own coordination cost, which is why most teams do it once during a launch crunch and never again. Annotation queues are LangSmith's answer to the logistics problem: a persistent, shared inbox of runs waiting for judgment, with the rubric, the assignment logic, and the output destinations built in. Once the pipeline exists, reviewing twenty traces a day becomes a habit rather than a project.

What Annotation Queues Are in LangSmith

An annotation queue in LangSmith is a curated list of runs — traces or individual spans from your tracing projects — waiting for human review. Think of it as a work queue in the classic sense: items go in from one side, reviewers pull items from the other, and each item leaves the queue once it has been handled. That framing sounds simple, but it carries several properties that make the difference between a workflow and a mess.

First, a queue is a separate object from your tracing project. You do not review runs by wandering through the firehose of all traces; you review the specific runs that were deliberately placed in the queue, whether by a person clicking a button or by an automation rule sampling production traffic. This separation is what makes review scalable — the queue is a filter on reality, sized to what your team can actually process.

Second, queues carry configuration that shapes the review itself. When you create a queue you give it a name and description, and crucially you can attach reviewer instructions — the rubric — and define which feedback keys reviewers should fill in, such as a numeric quality score, a binary correctness flag, or a categorical failure tag. You can also set a default dataset, so that when a reviewer decides a run is worth keeping as a test case, sending it there is one click.

Third, queues manage reviewer coordination. LangSmith supports multiple reviewers working the same queue concurrently. Runs are reserved while someone is reviewing them so two people do not silently duplicate effort, and reservations expire after a configurable timeout so an abandoned run returns to the pool. If you want redundancy for measuring inter-annotator agreement, you can require that multiple reviewers see each run before it is considered done.

Finally, everything a reviewer produces is recorded as structured feedback attached to the run. That means annotations are queryable later — you can filter runs by human score, export labeled examples, or compare human labels against automated evaluator scores on the exact same runs. The queue is not just a to-do list; it is the ingestion point for human ground truth.

Setting Up Your First Annotation Queue

Creating a queue takes a couple of minutes in the LangSmith UI. From the Annotation Queues section in the sidebar, create a new queue and give it a name that describes the review purpose, not just the app — "checkout-agent weekly quality review" beats "queue 1" when you have six queues a quarter from now. The description field is worth filling in properly because it is the first thing a new reviewer reads.

The two settings that deserve real thought are the reviewer instructions and the feedback configuration. Instructions are your rubric: what the reviewer is judging, what the score values mean, and what to do in ambiguous cases. Feedback configuration defines the actual fields reviewers fill in. Keep the number of fields small — one primary score plus one categorical tag plus free-text comments covers most needs. Every additional required field slows reviewers down and lowers throughput more than it raises signal.

You can also create and manage queues programmatically through the SDK, which is handy when you want queue setup to live in version control alongside the rest of your evaluation code:

from langsmith import Client

client = Client()

queue = client.create_annotation_queue(
    name="checkout-agent-review",
    description=(
        "Weekly human review of checkout agent traces. "
        "Score correctness 0/1, tag the failure mode, "
        "and add good examples to the regression dataset."
    ),
)

# Pull recent runs from the tracing project that look suspicious:
# negative user feedback, or unusually slow responses.
runs = client.list_runs(
    project_name="checkout-agent-prod",
    filter='and(eq(feedback_key, "user_score"), eq(feedback_score, 0))',
    limit=50,
)

client.add_runs_to_annotation_queue(
    queue_id=queue.id,
    run_ids=[run.id for run in runs],
)

This pattern — query runs by some criterion, push them into a queue — is the programmatic core of everything that follows. Once runs are in the queue, reviewers open it in the UI and start working through items one at a time, with the full trace visible: inputs, outputs, intermediate steps, retrieved documents, tool calls. Reviewing in the queue view rather than the raw trace view matters because the queue presents runs with your rubric alongside, keeps count of what remains, and advances automatically as each item is completed.

Feeding the Queue: Manual Adds, Automation Rules, and Sampling

A queue is only as good as what flows into it, and there are three ways to feed one. The first is manual: while browsing traces, anyone on the team can select a run and send it to a queue. This is perfect for the "huh, that's weird" moments — an engineer debugging something else spots an odd response and flags it for proper review instead of losing it. Encourage this habit; it costs seconds and captures exactly the anomalies automated sampling misses.

The second and most important method is automation rules. LangSmith lets you define rules on a tracing project that match runs by filter criteria and automatically perform actions on them, including adding them to an annotation queue. A rule has a filter, a sampling rate, and an action. This is where review strategy actually lives, because the filters you choose determine what your humans spend their attention on. Some patterns that earn their keep:

• Route all runs with negative end-user feedback (thumbs down, low rating) into the queue at 100 percent — these are your highest-information traces.
• Sample a small random percentage of all production traffic, say one to five percent depending on volume, so you maintain an unbiased view of typical quality rather than only reviewing disasters.
• Route runs where an online LLM-as-judge evaluator scored below a threshold — let the cheap automated judge do triage and reserve human attention for the cases it flags.
• Route runs with errors, unusually high latency, or suspiciously short outputs, which often indicate silent failures the user never reported.

The third method is programmatic, as in the SDK example earlier. This shines for one-off investigations — "queue every trace from the affected customer during yesterday's incident" — and for scheduled jobs that implement sampling logic more complex than rules support, such as stratified sampling across user tiers or intent categories.

Whatever mix you use, watch the queue depth. A queue that grows faster than reviewers drain it becomes demoralizing wallpaper within two weeks. It is far better to sample less and actually finish the queue every cycle than to accumulate ten thousand unreviewed runs that everyone learns to ignore. Queue size is a dial; turn it until inflow matches your team's honest review capacity.

Designing a Rubric Reviewers Can Actually Use

The rubric is where most annotation efforts quietly fail. Write one that is vague and every reviewer applies their private standards, producing labels that disagree with each other and teach you nothing. Write one that is a fifteen-field questionnaire and reviewers burn out by run thirty. The craft is in the middle.

Start from the decision the labels will drive. If the goal is "measure whether the agent resolves the user's issue," the primary field should be a binary resolved/not-resolved judgment, because binary questions produce far more consistent labels than ten-point scales. Humans are bad at agreeing on whether a response is a six or a seven; they are quite good at agreeing on whether it solved the problem. If you need granularity, use a small ordinal scale with anchored definitions — for instance, zero means factually wrong or harmful, one means technically correct but unhelpful, two means correct and helpful — and write those definitions into the reviewer instructions verbatim.

Add one categorical field for failure mode, with a short list of options you actually intend to act on: hallucination, retrieval miss, wrong tool call, formatting violation, refused unnecessarily, other. This field is what turns a pile of bad scores into an engineering backlog, because it tells you which subsystem to fix. Resist letting the category list grow past seven or eight options; long taxonomies produce noisy, lazy tagging.

Always include a free-text comment field and instruct reviewers to use it whenever they pick "other" or feel uncertain. The comments become the raw material for your next rubric revision, because they show you where the current one is failing to capture reality.

Then calibrate. Before letting the rubric loose, have two or three people independently annotate the same twenty runs and compare labels. Wherever they disagree, the rubric is ambiguous — discuss the disagreements, tighten the definitions, and re-run the exercise. Twenty minutes of calibration saves weeks of collecting labels you later realize you cannot trust. Repeat the spot-check periodically, especially after adding new reviewers, since annotation standards drift just like models do.

The Reviewer Workflow: What Annotating Actually Looks Like

Understanding the reviewer's experience matters because throughput and label quality both depend on it. When a reviewer opens a queue, LangSmith presents one run at a time: the input, the final output, and the full trace tree of intermediate steps for when the surface answer needs investigating. The feedback fields you configured sit alongside, together with your instructions. The reviewer reads, judges, fills in the fields, optionally leaves a comment, and moves to the next run. Runs they are viewing are reserved for them, so a colleague working the same queue is served different items.

For each run, the reviewer typically has three possible dispositions beyond scoring. They can mark it done, which records the feedback and removes it from the queue. They can add it to a dataset — this is the single most valuable button in the entire workflow, and we will come back to it. Or they can skip it when the run is outside their competence or the rubric genuinely does not apply, leaving it for someone else.

A few operational habits make the difference between review that sticks and review that fizzles. Timebox sessions: twenty to thirty minutes of focused annotation produces better labels than a two-hour slog, because judgment quality degrades noticeably with fatigue. Schedule it: a recurring thirty-minute slot where the on-call engineer or a rotating reviewer drains the queue beats "review when you have time," which reliably means never. Review recent runs: labels on last week's traffic inform decisions about the current system, while labels on three-month-old traces from a superseded prompt version are archaeology.

Also decide deliberately who reviews. Engineers catch technical failures — bad tool arguments, retrieval misses — but domain experts catch substantive ones. If your application answers tax questions, at least some fraction of reviewed runs should be seen by someone who knows tax, not just someone who knows LangChain. Queues make this practical because you can create separate queues with separate rubrics for technical review and domain review, fed by the same automation rules at different sampling rates.

From Annotations to Datasets and Aligned LLM Judges

Feedback scores sitting on runs are useful for dashboards, but the real compounding value of annotation queues comes from two downstream loops.

The first loop is dataset construction. Every time a reviewer encounters a run that represents something — a canonical success worth protecting, a nasty failure worth guarding against, an edge case nobody thought of — they add it to a dataset directly from the queue. For failure cases, the reviewer can correct the output before saving, so the dataset example carries the input paired with what the system should have said rather than what it did say. Over weeks, this produces the most valuable eval dataset you will ever own: one distilled from real production traffic, curated by human judgment, and continuously refreshed as your traffic evolves. Synthetic datasets and hand-written test cases have their place, but they encode what you imagined users would do. Annotation-sourced datasets encode what users actually do. When you later change a prompt or swap a model, running experiments against this dataset tells you whether you broke anything that has actually happened in the field.

The second loop is judge alignment. If you run LLM-as-judge evaluators — online over production traces or offline in experiments — you need to know whether the judge agrees with human judgment, and annotation queues generate exactly the paired data required. Because human feedback and evaluator feedback attach to the same runs under different feedback keys, you can pull both and measure agreement directly:

from langsmith import Client

client = Client()

runs = client.list_runs(
    project_name="checkout-agent-prod",
    filter='eq(feedback_key, "human_correctness")',
    limit=200,
)

agree, total = 0, 0
for run in runs:
    scores = {
        fb.key: fb.score
        for fb in client.list_feedback(run_ids=[run.id])
    }
    if "human_correctness" in scores and "judge_correctness" in scores:
        total += 1
        agree += scores["human_correctness"] == scores["judge_correctness"]

print(f"Judge/human agreement: {agree}/{total}")

When agreement is poor, read the disagreements — they tell you precisely how to revise the judge prompt, and the human label is the arbiter of who was right. Some teams formalize this into a standing "judge calibration" queue: a small random sample of judge-scored runs goes to humans every week, agreement is tracked over time, and a drop triggers a judge-prompt review. This is how you earn the right to trust automated evaluation at scale — not by assuming the judge is correct, but by continuously auditing it against people.

Scaling Review Across a Team

Once one queue works, the question becomes how to run several without the process rotting. A few structural patterns hold up well.

Separate queues by purpose, not by dumping everything into one. A triage queue fed by negative user feedback needs fast turnaround and a lightweight rubric. A calibration queue for judge alignment needs careful, unhurried labels. A dataset-building queue for a new feature needs domain experts. Mixing these in one queue forces one rubric to serve three jobs and reviewers to context-switch constantly.

Assign ownership. Every queue should have a named owner responsible for keeping inflow sane, rubric current, and the queue drained. Shared ownership of a queue works exactly as well as shared ownership of an inbox.

Use multi-reviewer settings deliberately. Requiring two reviewers per run doubles cost, so reserve it for queues where you are measuring agreement or where individual labels carry weight, such as ground truth for judge calibration. For routine triage, one reviewer per run is fine.

Track a small number of health metrics: queue depth over time, runs reviewed per week, and the distribution of failure-mode tags. The failure-mode distribution is the payoff — when "retrieval miss" jumps from ten percent of failures to thirty percent after an index change, the annotation pipeline has just caught a regression that no aggregate latency or cost metric would ever surface.

Finally, close the loop visibly. Reviewers keep reviewing when their labels demonstrably cause fixes, dataset growth, and prompt changes. Nothing kills an annotation program faster than the sense that labels vanish into a database nobody reads. A short monthly summary — what the queues found, what got fixed because of it — is cheap and keeps the flywheel spinning.

Common Mistakes to Avoid

A few failure patterns show up so often they are worth naming explicitly.

1. Queueing more than you can review. An overflowing queue is worse than a small one because it trains the team to ignore it. Cut sampling rates until the queue reliably hits zero each cycle.
2. Reviewing only failures. If everything in the queue comes from thumbs-down feedback, you will develop a distorted, catastrophizing picture of quality and no baseline for what normal looks like. Always blend in a random sample.
3. Vague rubrics. "Rate the quality 1-5" with no anchors produces labels that measure reviewer mood. Anchor every score value with a concrete definition and an example.
4. Skipping calibration. If you never measure whether two reviewers agree, you do not know whether your labels mean anything. Calibrate at the start and spot-check quarterly.
5. Letting labels be a dead end. If annotations never become dataset examples, judge corrections, or bug reports, the program is theater. Every review session should produce at least a few artifacts that outlive it.
6. Annotating stale traffic. Review recent runs from the current system version. Labels on outputs from a prompt you replaced last month answer questions nobody is asking.

None of these mistakes are exotic; they are the default outcomes if you set up a queue and hope. The teams that get durable value treat annotation as a product with users (the reviewers), an SLA (queue drained weekly), and outputs (datasets, calibrated judges, fixed bugs) — not as a checkbox.

Where to Go From Here

Annotation queues are the connective tissue between the two halves of LLM quality work: the automated half that scales infinitely but drifts, and the human half that stays grounded but cannot read ten thousand traces. Wired together properly — automation rules sampling production into queues, humans labeling against calibrated rubrics, labels flowing into datasets and judge alignment — you get an evaluation system that improves itself every week instead of decaying. Start small: one queue, one rubric with a binary score and a failure tag, one automation rule sampling a few percent of traffic, one recurring half-hour review slot. Within a month you will have a production-sourced eval dataset and a measured sense of how much to trust your automated scores, which is more than most teams ever build.

If you want to go deeper — tracing fundamentals, building datasets and experiments, online and offline evaluators, LLM-as-judge design, and the full human-feedback workflow covered in this article with hands-on projects — the LangSmith Tutorial course on teachyou.ai walks through the entire observability and evaluation stack step by step, from your first traced run to a production-grade review pipeline your whole team can operate.