LangSmith
LangSmith is LangChain's platform for tracing, evaluating, testing and monitoring LLM applications and agents, with or without LangChain. Learn its main features.
Overview
LangSmith is a commercial platform from LangChain for building, debugging, testing and monitoring applications that use large language models (LLMs). It records what happens inside an application, step by step, and gives teams tools to measure whether changes make it better or worse. It works with LangChain and LangGraph, but it also works with plain code and other frameworks through its SDKs and OpenTelemetry.
The need comes from how LLM applications fail. A wrong answer may start with a poor retrieval, a confusing prompt, a tool call with the wrong arguments or a model change. Without a record of every step, teams guess. LangSmith captures each step as a trace, then lets teams turn real examples into test datasets and run evaluations before and after each change.
An everyday analogy: a flight data recorder plus a flight simulator. The recorder (tracing) shows exactly what happened on a real flight. The simulator (datasets and experiments) lets you replay difficult situations with a new design before it carries passengers.
What it is
LangSmith groups its features into a few product areas.
| Area | What it does | Typical use |
|---|---|---|
| Observability | Captures traces of LLM calls, tool calls, retrieval and custom functions; dashboards and alerts | Debugging, latency and cost analysis, production monitoring |
| Evaluation | Datasets of examples, evaluators, experiments and comparisons | Regression testing before release |
| Human feedback | Annotation queues and feedback scores on runs | Expert review, labelling ground truth |
| Online evaluation | Evaluators that run automatically on samples of production traces | Catching quality drift |
| Prompt engineering | Versioned prompts and a playground for trying models and prompts | Iterating on prompts with the team |
| Deployment | Hosting for agents, mainly built with LangGraph | Running agents as managed services |
LangSmith is offered as a managed cloud service. Enterprise customers can also run it in their own cloud account, with LangChain managing the control plane, or fully self-hosted.
Where it came from
LangChain started in 2022 as an open-source framework by Harrison Chase for composing LLM applications. As people moved LangChain apps to production, debugging and testing became the main pain. The company launched LangSmith in closed beta in mid-2023 as a companion platform for tracing and evaluation.
In February 2024, LangChain announced that LangSmith was generally available, together with a Series A round led by Sequoia Capital. Since then the platform has added online evaluation, annotation queues, prompt versioning, OpenTelemetry ingestion, multi-turn thread views for agents and deployment features for LangGraph agents. Product names and plans change often, so check the current documentation for details.
How it works
1. Instrument the application
You install the SDK (Python or TypeScript) and set an API key. With LangChain or LangGraph, setting environment variables such as LANGSMITH_TRACING=true is enough to send traces. For other code, you wrap functions with the traceable decorator and wrap model clients so that each call is recorded. Apps already instrumented with OpenTelemetry can send spans to LangSmith's OTLP endpoint instead.
from langsmith import traceable
from langsmith.wrappers import wrap_openai
import openai
client = wrap_openai(openai.OpenAI())
@traceable
def answer(question: str) -> str:
r = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": question}])
return r.choices[0].message.content
2. Inspect traces
Each request becomes a trace: a tree of runs. A run has inputs, outputs, timing, token counts, errors and metadata. Traces live in projects, so production and development stay separate. You can filter, search and open any run to see the exact prompt the model received.
3. Build datasets
You collect examples: inputs plus, where available, reference outputs. Good sources are real traces that went wrong, edge cases written by experts and synthetic variations. Datasets are versioned, so experiments stay comparable.
4. Define evaluators
Evaluators score an output. They can be code (exact match, JSON validity, a regex), an LLM as judge with a rubric, or human review. Each returns a score and, ideally, a short reason.
5. Run experiments
An experiment runs your application over a dataset and applies the evaluators. You then compare experiments side by side: the old prompt against the new one, or one model against another. Differences show up per example, so you can see which cases improved and which regressed.
6. Monitor production
Dashboards track volume, latency, errors, token use and feedback. Online evaluators score a sample of live traffic. Rules can route interesting traces, such as those with negative user feedback, into datasets or annotation queues. The loop then repeats.
Key concepts and terms
| Term | Meaning |
|---|---|
| Run | One recorded step, such as an LLM call or a tool call |
| Trace | The tree of runs for one request |
| Project | A container for traces from one application or environment |
| Thread | A sequence of traces forming one multi-turn conversation |
| Dataset | A versioned set of examples used for testing |
| Example | One input, with an optional reference output |
| Evaluator | A function, model or person that scores outputs |
| Experiment | A run of the application over a dataset, with scores |
| Feedback | A score or comment attached to a run |
| Annotation queue | A list of runs waiting for human review |
| Online evaluation | Automatic scoring of live production traces |
A worked example
This is an illustrative scenario.
A team runs a support assistant that answers billing questions using retrieval over 4,000 help articles. After a model upgrade, complaints rise.
- They filter production traces for runs with a thumbs-down score in the past week and find 180.
- Opening a sample, they see that many failures start in retrieval: the top chunks are about a retired pricing plan.
- They add 60 of these cases, plus 40 normal cases, to a dataset of 100 examples, with correct answers written by a support lead.
- They define three evaluators: a code check that cited article ids exist, an LLM judge for correctness against the reference, and a judge for tone.
- Experiment A (current system) scores 71 of 100 correct. They filter retired articles from the index and run experiment B, which scores 88 of 100. Three examples regress, so they inspect those before release.
- After release, an online evaluator scores 5% of live traffic for correctness, and a rule sends low scores to an annotation queue.
Strengths and limitations
Strengths:
- Detailed traces of complex chains and agents, with a clear tree view.
- Evaluation, datasets, human review and monitoring in one place.
- Very simple setup for LangChain and LangGraph users.
- Works with other frameworks through SDKs and OpenTelemetry.
- Managed service, with enterprise options for private hosting.
Limitations:
- Commercial product; the platform itself is not open source.
- Self-hosting and bring-your-own-cloud require an enterprise plan.
- Tracing sends prompts and outputs to the platform, which needs a data-privacy review.
- Costs grow with trace volume and retention.
- LLM-judge scores need calibration against human judgement.
How it compares with alternatives
| Tool | Licence | Strength | Consider it when |
|---|---|---|---|
| LangSmith | Commercial | Deep LangChain and LangGraph integration | You use LangChain tools or want a managed suite |
| Langfuse | Open-source core | Self-hosting and open data model | You need to run it yourself without an enterprise plan |
| Phoenix | Open source | OpenTelemetry-native tracing and evals | You want a local or self-hosted tool built on open standards |
| Braintrust | Commercial | Evaluation-first workflow | Experiments and scoring are your main focus |
| MLflow | Open source | Broader ML lifecycle | You already track classic ML models in MLflow |
Practical guidance and best practices
- Trace from day one, including development, so you have examples when problems appear.
- Add metadata such as user tier, prompt version and model name to every trace.
- Keep separate projects for development, staging and production.
- Turn real failures into dataset examples; they are worth more than invented ones.
- Start with simple code evaluators, then add LLM judges for qualities code cannot check.
- Calibrate LLM judges against a sample of human labels before trusting them.
- Run an experiment before every prompt or model change and compare per example.
- Mask or omit sensitive fields before they leave your application.
- Sample production traffic for online evaluation to control cost.
Common mistakes
- Only looking at aggregate scores and missing the examples that regressed.
- Building datasets entirely from synthetic examples that do not resemble real traffic.
- Trusting an LLM judge without checking it against people.
- Logging personal data in traces without a privacy plan.
- Changing the dataset between experiments and then comparing scores.
Frequently asked questions
Do I need LangChain to use LangSmith?
No. LangSmith works with any code through its Python and TypeScript SDKs, the traceable decorator and client wrappers, and it accepts OpenTelemetry traces. LangChain and LangGraph simply need less setup.
Can LangSmith be self-hosted?
Yes, for enterprise customers. Options include fully self-hosted deployments and a bring-your-own-cloud model where data stays in your cloud account.
What is the difference between a run and a trace?
A run is one step, such as a single model call. A trace is the whole tree of runs that make up one request, from the first input to the final answer.
How does LangSmith evaluate outputs?
You provide a dataset and one or more evaluators. Evaluators can be code checks, LLM judges or human reviewers. An experiment runs the app over the dataset and records every score so you can compare versions.
Is LangSmith only for development?
No. It is also used in production for monitoring, alerts, online evaluation and collecting user feedback.
What data does LangSmith store?
By default it stores the inputs, outputs and metadata of traced steps, including prompts and responses. You can mask or skip fields and set retention, but plan this before tracing sensitive workloads.
Glossary
- LLM: a large language model that generates text.
- Observability: being able to understand a system's behaviour from its outputs and records.
- Span: an OpenTelemetry unit of work, similar to a run.
- Regression: a change that makes some results worse.
- Reference output: the expected answer for a test example.
- Rubric: written criteria a judge uses to score answers.
- Playground: an interface for trying prompts and models interactively.
Summary
LangSmith is LangChain's platform for observing, evaluating and improving LLM applications and agents. It records every step as a trace, turns real cases into datasets, scores outputs with code, model and human evaluators, and monitors production quality. It is easiest with LangChain and LangGraph but works with any stack. Plan for data privacy and cost, calibrate judges against people, and compare experiments example by example. See evals for the wider practice of evaluation.
Related
It belongs to LLM observability and evaluation in my stack, under Generative AI and LLMs. Others in the same group: