LangSmith Onboarding: Getting Your Team Set Up in a Day

Most teams treat LangSmith onboarding as an afterthought. Someone signs up with a personal email, pastes an API key into a notebook, sees a trace appear, and declares the tool "set up." Three weeks later there are four half-abandoned projects named default, test, test2, and final-test, nobody knows which API key is in production, and the one person who created the account is on holiday while the rest of the team stares at a login screen. The tool works — the setup around it does not. The good news is that a clean, team-wide LangSmith onboarding genuinely fits in a single working day if you do things in the right order. This guide walks through that order: organization and workspace structure in the morning, key management and tracing before lunch, roles, datasets, and alerts in the afternoon. Follow it top to bottom and by the end of the day every engineer on your team will be sending traces, reading them, and running their first evaluation.

Understand What You Are Actually Setting Up

Before clicking anything, it helps to know the three layers LangSmith organizes everything into, because most onboarding mistakes are really structure mistakes.

At the top is the organization. This maps to your company or business unit. Billing lives here, and so does membership at the highest level. You want exactly one organization per company in almost every case.

Inside an organization sit workspaces. A workspace is the unit of isolation: projects, datasets, prompts, annotation queues, and dashboards all live inside a workspace, and they are not visible across workspace boundaries. Members are invited to the organization and then assigned to one or more workspaces. A sensible default for a small-to-medium team is two workspaces: one for development and experimentation, one for production. Larger companies often cut workspaces per product line instead, then separate dev and prod with projects inside each.

Inside a workspace sit tracing projects. A project is a bucket of traces. Every run your application sends is filed into a project, either the one named in your environment configuration or a project called default if you forgot to set one. Projects are cheap — create them freely. Datasets and prompts, by contrast, are workspace-level resources shared across projects, which is exactly what you want: a regression dataset built from production traces should be usable in your dev experiments.

There is one more distinction worth settling before you begin: cloud versus self-hosted. The SaaS offering at smith.langchain.com is the fastest path and is what this guide assumes. There is a separate EU data-residency region, and an Enterprise self-hosted option that runs in your own Kubernetes cluster for teams with strict data-control requirements. If you are in a regulated industry, decide this today, not after a month of traces has landed in the wrong region — traces do not migrate between regions.

One caution on data hygiene while we are here: traces capture full inputs and outputs by default. That means user messages, retrieved documents, and model responses. Decide on day one whether that is acceptable for your data classification, and if not, plan to scrub or anonymize fields before they leave your service. Doing this later means re-plumbing every integration point.

Hour One: Organization, Workspaces, and Naming Conventions

Start with a clean slate. Create the organization with a shared or role-based email if your company has one, not a personal address — you do not want the org rooted to an individual who might leave. Once the organization exists, upgrade to a plan that supports multiple seats. The free developer tier is single-seat and fine for evaluation, but a team onboarding needs a team plan so everyone can log in under their own identity. Do not share one login across five engineers; you lose the audit trail and you will fight session conflicts constantly.

Next, create your workspaces. For most teams starting out:

• acme-dev — every engineer has broad permissions here; experiments, prototypes, and prompt drafts live in this workspace
• acme-prod — production traces land here; write access is limited to service keys, and human members mostly read

Then agree on project naming before anyone sends a trace. This sounds bureaucratic; it is the single highest-leverage ten minutes of the day. A convention that works well is service-environment, for example support-bot-staging, support-bot-prod, rag-pipeline-dev. When you have forty projects in six months, a grep-able convention is the difference between finding things and archaeology.

Finally, agree on a small set of standard metadata keys you will attach to runs — things like git_sha, prompt_version, customer_tier, and feature_flag. LangSmith lets you filter and group traces by metadata, but only if people attach it consistently. Write these conventions in your team wiki now, while the slate is empty.

Hour Two: API Keys Done Properly

LangSmith has two kinds of credentials, and mixing them up is the most common onboarding mistake worth preventing.

Personal Access Tokens (PATs) are tied to a human user. They act as that user, inherit that user's permissions, and die when that user is removed from the organization. They are right for local development and for individual scripts.

Service keys are tied to the workspace itself, not to any person. They keep working when people leave. They are the only kind of credential that should ever appear in a deployed environment, a CI pipeline, or a shared secret store.

The rule to write down: humans get PATs, machines get service keys. Concretely, during hour two you should mint:

1. One service key for the production deployment, stored in your secret manager (Vault, AWS Secrets Manager, Doppler — whatever you already use)
2. One service key for CI, scoped to the dev workspace, used by your evaluation jobs
3. Nothing else — each engineer creates their own PAT when they onboard themselves

Keys are shown once at creation, so store them immediately. Never commit them; add your env files to .gitignore before anyone writes one. And schedule a calendar reminder now to rotate the service keys — quarterly is a reasonable default. Rotation is painless when you planned for it and miserable when the key is hardcoded in six repositories.

While you are in the settings screens, also check your data retention configuration. Traces have a base retention tier and an extended one, and the right mix affects both cost and how far back your debugging can reach. Make an explicit choice rather than accepting whatever default happens to apply.

Hour Three: Wire Up Tracing With Environment Variables

Now the part everyone actually came for. LangSmith tracing is controlled almost entirely by environment variables, which is what makes rollout across services so quick. The canonical set:

# .env — never commit this file
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="lsv2_pt_your_key_here"
export LANGSMITH_PROJECT="support-bot-dev"
export LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
# EU region teams use: https://eu.api.smith.langchain.com

A few notes that save debugging time. First, you may see older tutorials using LANGCHAIN_TRACING_V2 and LANGCHAIN_API_KEY — those legacy names still work, but standardize on the LANGSMITH_ prefix so your team's dotfiles all look the same. Second, LANGSMITH_PROJECT is what routes traces to the right project; if traces are mysteriously landing in default, this variable is unset or your process manager is not passing it through. Third, the endpoint variable only matters if you are on the EU instance or self-hosted — but set it explicitly anyway, because explicit beats implicit when a new hire copies your env template.

Install the SDK in whatever stack you run:

pip install -U langsmith           # Python
npm install langsmith              # TypeScript / JavaScript

If your application is built on LangChain or LangGraph, you are already done — with those environment variables set, every chain, agent step, tool call, and retriever invocation is traced automatically with no code changes. This is the single biggest onboarding shortcut and the reason hour three is usually only twenty minutes.

If you are not using LangChain, you decorate the functions you care about. The @traceable decorator and the OpenAI wrapper cover most plain-Python applications:

from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI

client = wrap_openai(OpenAI())  # every completion call is now traced

@traceable(run_type="retriever")
def retrieve_docs(query: str) -> list[str]:
    return vector_store.search(query, k=4)

@traceable(name="answer_question", metadata={"prompt_version": "v3"})
def answer_question(question: str) -> str:
    docs = retrieve_docs(question)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": f"Context:\n{'\n'.join(docs)}"},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content
</code_omitted>

The nesting happens automatically: because retrieve_docs is called inside answer_question, it appears as a child run in the trace tree, and the wrapped OpenAI call appears as a sibling with token counts, latency, and cost attached. That tree view — retrieval step, LLM call, output, each with its own timing — is the core value of the whole platform, and your team gets it from roughly fifteen lines of decoration.

Have every engineer run one traced request from their own machine before lunch, open the project in the UI, and find their trace. Onboarding sticks when each person sees their own code appear in the dashboard, not when they watch someone else's screen share.

Hour Four: Invite the Team and Assign Roles

With the plumbing proven, bring everyone in. Invitations happen at the organization level; workspace assignment happens per workspace. Resist the urge to make everyone an admin because it is Tuesday and you are busy.

A role layout that works for most teams:

• Two organization admins. Not one — an org with a single admin has a bus-factor problem; the second admin exists for the day the first is unreachable. Not five — admin sprawl means nobody knows who changed the billing plan.
• Engineers as workspace members in `acme-dev` with full ability to create projects, datasets, prompts, and experiments.
• Read-mostly access in `acme-prod`. Engineers debug production by reading traces, annotating them, and adding them to datasets. Actual write traffic to prod projects comes from the service key, not from humans.
• Non-engineers get seats too. This is the underrated part of LangSmith onboarding: product managers and domain experts can review traces and label outputs in annotation queues without touching code. If a subject-matter expert will be judging answer quality — and they should — invite them today and show them the annotation view.

If you are on an enterprise plan, custom RBAC roles let you get finer-grained — for example, a role that can annotate but not delete, for external contractors. Most teams do not need this on day one, but knowing it exists prevents you from contorting the workspace layout to fake permissions.

Close the hour with a fifteen-minute tour for everyone: how to find a trace by filtering on metadata, how to read the run tree, where latency and token costs show up, and how to share a trace link in Slack when asking for help. A shared trace URL that a teammate can open and inspect replaces the "can you paste the full prompt and the output and the stack trace" ritual permanently.

Hour Five: Build Your First Dataset From Real Traces

Tracing tells you what happened; datasets and evaluations tell you whether it was any good. Do not leave day one without at least a small dataset, because the habit of curating examples is the thing that separates teams who improve their LLM systems from teams who vibe-check them.

The fastest path is to harvest the traces you generated this morning. In the project view, select a handful of representative runs — a few good ones, a few embarrassing ones — and add them to a new dataset. Each example stores the inputs and, where you have one, the expected output. Ten examples is a perfectly respectable day-one dataset; you will grow it every time production surprises you.

You can also seed examples programmatically, which is worth doing once so the team sees the SDK side:

from langsmith import Client

client = Client()
dataset = client.create_dataset(
    dataset_name="support-bot-regression",
    description="Golden questions the support bot must never regress on",
)
client.create_examples(
    dataset_id=dataset.id,
    examples=[
        {
            "inputs": {"question": "How do I reset my password?"},
            "outputs": {"answer": "Go to Settings > Security > Reset Password."},
        },
        {
            "inputs": {"question": "Do you offer refunds after 30 days?"},
            "outputs": {"answer": "Refunds are available within 30 days of purchase only."},
        },
    ],
)

Name datasets by purpose, not by date: support-bot-regression, rag-hallucination-cases, tone-of-voice-golden. A dataset named test-data-june tells future-you nothing.

Then run one evaluation, even a trivial one, so the loop is closed end to end:

from langsmith import Client

client = Client()

def correctness(outputs: dict, reference_outputs: dict) -> bool:
    return reference_outputs["answer"].lower() in outputs["answer"].lower()

results = client.evaluate(
    lambda inputs: {"answer": answer_question(inputs["question"])},
    data="support-bot-regression",
    evaluators=[correctness],
    experiment_prefix="day-one-baseline",
)

The evaluator here is deliberately naive — a substring check — because the point today is the workflow, not the metric. Your team sees an experiment appear in the UI, each example scored, each run traced, and a comparison view ready for the day you change a prompt and want to know if it helped. Swap in an LLM-as-judge evaluator next week; establish the muscle memory now.

Hour Six: Dashboards, Alerts, and Cost Visibility

The last technical block of the day is making sure LangSmith tells you when something goes wrong, rather than waiting for you to look.

Start with the built-in project dashboards. Every tracing project gets charts for trace volume, error rates, latency percentiles, and token usage out of the box. Walk the team through where P99 latency lives, because "the bot feels slow" becomes a much shorter conversation when anyone can pull up the percentile chart and the slowest traces behind it.

Then configure alerts on the production project. The three that earn their keep immediately:

1. Error rate — alert when the percentage of failed runs in a window crosses a threshold you choose; this catches provider outages and broken deployments
2. Latency — alert on P99 crossing your budget, which surfaces retrieval slowdowns and model provider degradation before users file tickets
3. Feedback score drops — once you log user feedback (thumbs up/down mapped to feedback scores on runs), alert on a falling average; this is your earliest signal of quality regression that no infrastructure metric will ever catch

Route alerts to the channel your team actually reads — a webhook into Slack or PagerDuty — not to the email inbox nobody opens. And assign each alert an owner. An alert without an owner is a notification, and notifications get muted.

Cost visibility deserves five minutes of its own. Because traces carry token counts and model names, LangSmith can attribute spend per project and, if you attached metadata like customer_tier this morning, per segment. Show the team where this lives. The first time someone discovers that one internal test harness is producing a third of the token bill, the whole day pays for itself.

Common Onboarding Pitfalls and How to Dodge Them

A short field guide to the mistakes that cost teams their second week:

• Everything in one project. If dev experiments and production traffic share a project, every metric is polluted and every filter is a chore. Separate environments at minimum; separate services ideally.
• Personal API keys in production. It works right up until that person's account is deactivated, and then production tracing silently stops. Service keys for machines, always.
• No sampling plan for high-volume services. Tracing every request is right for day one and often wrong at scale. Know that the SDK supports sampling via LANGSMITH_TRACING_SAMPLING_RATE, and decide your rate deliberately when volume grows rather than discovering the ingestion bill.
• Tracing secrets. Inputs and outputs are stored verbatim unless you intervene. If prompts can contain PII or credentials, add scrubbing before wide rollout, not after your first audit.
• Datasets nobody maintains. A dataset is a living artifact. Adopt the rule that every production incident adds at least one example to a regression dataset. Six months of that habit produces the most valuable QA asset your AI team owns.
• Skipping the non-engineers. If quality judgments live only in engineers' heads, your evaluations encode engineer taste, not user needs. Annotation queues exist precisely so domain experts can label outputs; onboard them like first-class users.
• Blocking the request path. Trace submission is asynchronous and batched by design, but if you build custom flush logic — for example in short-lived serverless functions — make sure you flush on shutdown without blocking user responses. Test this before it tests you.

None of these are hard to avoid. All of them are hard to unwind once three services and forty projects deep.

Your End-of-Day Checklist

By the time you close the laptop, the team should be able to tick every box on this list:

1. One organization, on a multi-seat plan, with two admins
2. Dev and prod workspaces created, with a written project-naming and metadata convention
3. Service keys in the secret manager for production and CI; every engineer holding their own personal token
4. Environment variables templated in every service repo, with tracing verified end to end by each engineer individually
5. At least one non-LangChain code path decorated with @traceable, or automatic tracing confirmed for your LangChain/LangGraph stack
6. A named regression dataset with ten or more real examples pulled from today's traces
7. One baseline experiment run and visible in the experiments view
8. Error-rate, latency, and feedback alerts pointing at a channel with a named owner
9. A wiki page recording all of the above, so the next hire onboards in an hour instead of a day

That is a genuinely complete LangSmith onboarding: not just an account that exists, but a team that traces by default, evaluates before shipping, and gets paged by quality signals instead of by customers.

If you want to go deeper than setup — building rigorous LLM-as-judge evaluators, wiring evaluations into CI so prompt changes are gated like code changes, designing annotation workflows for subject-matter experts, and running online evaluations against live production traffic — that is exactly what we teach in the LangSmith Tutorial course on teachyou.ai. It picks up where this one-day setup ends and takes your team from "we can see our traces" to "we ship LLM changes with the same confidence we ship code." Your future self, staring at a P99 chart at 2 a.m. with a shareable trace link already in hand, will thank you for starting today.