Braintrust

Overview

Braintrust is a commercial platform for evaluating and improving applications built on large language models (LLMs) and agents. Its centre of gravity is the evaluation, or "eval": you take a set of example inputs, run your application on them, score the outputs and compare the scores with earlier versions. Around that core it adds production logging, a prompt playground, versioned datasets and an AI assistant that helps analyse results.

The idea behind it is that AI products should be developed like software with tests. A prompt tweak or a model upgrade can quietly make some answers worse. Braintrust makes those changes measurable by running the same examples and scorers every time and showing what improved and what regressed.

An everyday analogy: a cooking school's tasting panel. Every new version of a recipe is cooked from the same ingredients, tasted by the same panel using the same score sheet, and compared with last week's version. Nobody argues from memory about whether it got better; the score sheets show it.

What it is

Braintrust can be used as a hosted service, and it supports a hybrid deployment in which the data plane runs in the customer's own cloud.

Where it came from

Braintrust was introduced in September 2023 by Ankur Goyal. He had previously led engineering at MemSQL, later SingleStore, and founded Impira, which joined Figma in 2022. Working on AI features there, he saw how hard it was to know whether changes to prompts and models helped, and built internal eval tooling that became the starting point for Braintrust.

Hybrid deployment, keeping customer data in the customer's cloud, was supported from the start. Later milestones include deployable functions for prompts, tools and scorers in October 2024, and Brainstore, a database written in Rust for AI logs, which became the default storage for the hosted product in March 2025. The Loop assistant became available to all users in 2025. The company has raised venture funding from investors including Greylock, Andreessen Horowitz and ICONIQ.

How it works

1. Write an eval

An eval has three parts: data, a task and scores. The task is the code under test, often a call to your application. Scores are functions that take the input, output and expected value and return a number between 0 and 1.

from braintrust import Eval
from autoevals import Levenshtein

Eval(
    "greeting-bot",
    data=lambda: [{"input": "Ada", "expected": "Hi Ada"}],
    task=lambda input: "Hi " + input,
    scores=[Levenshtein],
)

Running the file with the Braintrust CLI executes the task on every example, applies the scorers and uploads the results as an experiment.

2. Compare experiments

The UI shows each experiment's average scores and, more usefully, a per-example diff against a baseline. You can sort by the examples whose scores dropped, open their traces and read the inputs, outputs and intermediate steps.

3. Build better datasets

Datasets are versioned, so an experiment always points at the exact examples it used. You can add examples by hand, upload them, or promote interesting production logs into a dataset.

4. Log production

The same SDK records production requests as traces with spans for model calls, retrieval and tools. Because logs and evals share a format, a poor production answer can become a test case with a click, and online scorers can rate a sample of live traffic.

5. Iterate in the playground

Product managers and domain experts can try prompt or model changes in the playground against a dataset and see scores update, without writing code. Saved prompts are versioned and can be called from the application.

6. Use AI help

Loop, the built-in assistant, can search logs, draft scorers, suggest dataset examples and summarise why an experiment scored differently. Its suggestions still need checking.

Key concepts and terms

A worked example

This is an illustrative scenario.

A travel company's assistant writes short itinerary summaries. The team wants to switch to a cheaper model without losing quality.

1. They assemble a dataset of 250 real requests from logs, with summaries approved by editors as expected outputs.
2. They define four scorers: a JSON-format check in code, a length check (under 120 words), an LLM judge for factual consistency with the booking data, and an autoevals similarity scorer against the editor's version.
3. Experiment A uses the current model: format 100%, length 96%, factual 0.91 average.
4. Experiment B uses the cheaper model: format 99%, length 98%, factual 0.86.
5. Sorting by regressions shows 19 cases where the cheaper model dropped flight times. They add an instruction to the prompt and run experiment C: factual 0.90, with only 4 regressions left.
6. They ship C behind a flag and enable online scoring on 10% of traffic to confirm the result holds.

The decision is based on per-example evidence, and the dataset now protects against the same regression in future.

Strengths and limitations

Strengths:

• Evaluation-first design with clear per-example comparisons.
• Same format for evals and production logs, so real failures become tests easily.
• Usable by engineers in code and by non-engineers in the playground.
• Open-source autoevals library for common scorers.
• Hybrid deployment option for data residency.

Limitations:

• Commercial platform; the core service is not open source.
• Costs and plans change, so budgets need checking against current pricing.
• Scores are only as good as the scorers and datasets you design.
• LLM-judge scorers add model cost and need calibration.
• Tracing sends prompts and outputs to the platform unless you use hybrid deployment.

How it compares with alternatives

Practical guidance and best practices

1. Start with 30 to 100 real examples rather than waiting for a perfect dataset.
2. Write simple code scorers first: format, length, required fields.
3. Add LLM-judge scorers for qualities code cannot check, and keep their rubrics short and specific.
4. Check judge scores against human ratings on a sample before trusting them.
5. Always compare against a named baseline and read the regressed examples, not only the averages.
6. Promote production failures into datasets every week.
7. Run evals in CI so prompt and model changes are tested before merge.
8. Use metadata and tags to compare results by user segment or feature.

Common mistakes

• Optimising for an average score while specific important cases regress.
• Using only synthetic examples that do not reflect real requests.
• Letting the dataset change silently between experiments.
• Writing vague judge rubrics that produce noisy scores.
• Treating a single run as final when the model output is non-deterministic.

Frequently asked questions

What is an eval in Braintrust?

It is a repeatable test made of three parts: a dataset of examples, a task function that produces outputs, and scorers that rate those outputs. Running it creates an experiment you can compare with earlier ones.

Is Braintrust open source?

The platform is commercial. Braintrust publishes an open-source library of scorers called autoevals, and its SDKs are available for Python and TypeScript.

Can I use Braintrust for production monitoring?

Yes. It logs production traces in the same format as evals, supports online scoring of sampled traffic, and lets you turn logged cases into dataset examples.

Can my data stay in my own cloud?

Braintrust supports a hybrid deployment in which the data plane runs in your cloud account while the hosted service provides the interface. Check current plans for availability.

How do scorers work?

A scorer receives the input, the output and optionally the expected output, and returns a score between 0 and 1. Scorers can be deterministic code, an LLM judge with a rubric or a human rating.

Who uses the playground?

Anyone on the team. It lets non-engineers such as product managers or subject experts test prompt and model changes against datasets and see the scores.

Glossary

• Regression: a change that makes some outputs worse.
• Rubric: written criteria a judge uses to score output.
• Non-deterministic: giving different outputs for the same input.
• Span: one step inside a trace.
• CI: continuous integration, automatic tests run on code changes.
• Data plane: the part of a system that stores and processes customer data.
• Levenshtein distance: the number of edits needed to turn one string into another.

Summary

Braintrust is an evaluation-first platform for AI applications. It ties together datasets, task functions, scorers and experiments, and links them to production logs and a prompt playground so that every change can be measured case by case. It is commercial, with a hybrid option for data residency and an open-source scorer library. Use real examples, simple scorers first, calibrated judges and named baselines, and read the regressions rather than only the averages. See evals and LLM as judge for the methods behind it.

Related

It belongs to LLM observability and evaluation in my stack, under Generative AI and LLMs. Others in the same group: