TruLens

Overview

TruLens is an open-source Python library for evaluating and tracing applications built on large language models. It records what an app does on each run, such as the input, the retrieved context, the intermediate calls and the final answer. It then scores those records with small programs called feedback functions. The scores tell you whether an answer is grounded in the retrieved text, whether the retrieved text is relevant to the question, and whether the answer actually addresses the question.

The main reason to use TruLens is that LLM apps fail in quiet ways. A retrieval pipeline can return the wrong passages and the model can still write a fluent answer. TruLens makes those failures visible and measurable. You can compare versions of a prompt, a retriever or a model on the same inputs, and pick the version with better scores. It pairs naturally with evals, tracing and regression boards, and it is often used next to RAGAS and DeepEval.

What it is

TruLens is a developer library. You wrap your application, run it on test inputs or live traffic, and TruLens stores a record of every call in a local or remote database. A dashboard lets you browse the records, read the traces and compare the scores of different app versions.

It has three jobs.

• Instrumentation. It attaches to your app code and captures the inputs, outputs and intermediate steps of each call.
• Evaluation. It runs feedback functions over those captured records and stores the scores.
• Comparison. It groups runs by app version so you can see which change helped and which one hurt.

What it is not

TruLens is not a model, not a vector store and not a prompt framework. It does not generate answers. It watches the pipeline that does and judges the results. It also does not replace human review. It gives you a scalable first pass and a way to find the cases worth reading by hand.

Typical users

• Teams that build retrieval-augmented generation (RAG) apps and need to know if the answers are supported by the sources.
• Teams that build agents and need a trace of each step and tool call.
• Teams that tune prompts, chunk sizes or retrievers and want numbers rather than impressions.

Where it came from

TruLens grew out of the need to evaluate LLM apps once they moved from demos to production around 2023. Early LLM work relied on a few hand-picked examples and on reading outputs by eye. That does not scale. Teams needed something closer to the test suites and monitoring dashboards used in normal software.

The project introduced the idea of feedback functions as a general, named concept: a function that takes pieces of a run and returns a score, usually between zero and one. It also popularised the "RAG triad" of context relevance, groundedness and answer relevance as a compact way to check a retrieval pipeline.

Later versions moved toward standard tracing ideas, so that traces from an app can follow the same shape used in the wider observability world, as seen in OpenTelemetry. The exact API has changed between releases, so check the documentation for the version you install.

Related ideas it builds on

• Model-graded evaluation. Using one language model to score another model's output. See LLM as judge.
• Software testing. Fixed test inputs, expected behaviour and regression checks.
• Observability. Logs, metrics and traces that explain what a running system did.

How it works

Think of TruLens as a flight recorder plus a panel of inspectors. The recorder captures everything that happens during a flight. The inspectors read the recording afterward and score it on specific points: did the plane follow the route, was the landing smooth, was the fuel use normal. You can add or remove inspectors without changing the plane.

Step 1: Wrap the app

You build your app as usual, with a retriever, a prompt and a model. Then you wrap it with a TruLens recorder. The wrapper hooks into the methods that matter, such as the retrieval method and the generation method, and records their inputs and outputs.

Step 2: Define feedback functions

A feedback function says what to score and from which parts of the record. For a RAG app you usually choose three.

• Context relevance. How relevant is each retrieved chunk to the question?
• Groundedness. Are the claims in the answer supported by the retrieved chunks?
• Answer relevance. Does the answer address the question that was asked?

Each function is told where to find its inputs in the record. In TruLens this is done with selectors that point to the question, the retrieved context and the final answer.

Step 3: Run the app under the recorder

You call the app inside a recording context. Each call creates a record. The record holds the trace of steps, timing, any cost and token information the integration exposes, and the inputs and outputs.

Step 4: Score the records

The feedback functions run on the records. Many of them call a language model as a judge. The judge gets a prompt such as "rate how well this statement is supported by this text" and returns a number, often with a short reason. Other feedback functions use embeddings, small classifiers or plain code.

Step 5: Compare and iterate

The dashboard shows average scores per app version and lets you drill into the lowest scoring records. You change a prompt, a chunk size or a retriever setting, run again as a new version, and compare.

An everyday analogy for groundedness

Imagine a student writing an essay from a set of library books. Groundedness asks: for every claim in the essay, can you point to a sentence in the books that backs it up? A fluent essay full of claims with no source would score low. This is the check that catches hallucination in a RAG app.

An everyday analogy for context relevance

Imagine asking a librarian for books about bread baking and receiving a pile that includes cookbooks, a novel and a car manual. Context relevance scores how much of the pile is useful for the question. A low score points to the retriever, not the language model.

How groundedness is usually scored

A common method splits the answer into separate statements. For each statement, a judge model looks at the retrieved context and decides how well the context supports it. The scores are then combined, often by averaging or by taking the minimum. Some versions also ask the judge to give a short reason, which helps when you read failures.

Key concepts and terms

Feedback functions in more detail

A feedback function has three parts: a provider, a prompt or method, and a set of inputs. The provider is the thing doing the judging, such as a hosted language model or a local one. The prompt or method is the scoring rule. The inputs are the selected parts of the record.

Feedback functions can be run in different modes. Some run right after each call. Others run later in a batch so that slow judge calls do not slow the app.

Reference-free and reference-based scores

• Reference-free scores need no ground truth. Groundedness and context relevance are of this kind. They only need the question, the context and the answer.
• Reference-based scores compare the answer to a known correct answer. They need a labelled test set.

Reference-free scores are handy for live traffic, where no labels exist. Reference-based scores are better for a fixed regression set.

A worked example

This example sketches a small RAG app with TruLens feedback. The class and function names change across versions, so treat it as a pattern and not an exact script.

# Illustrative pattern; check the docs for your installed version.
from trulens.core import TruSession, Feedback, Select
from trulens.apps.app import TruApp, instrument
from trulens.providers.openai import OpenAI as JudgeProvider

session = TruSession()
session.reset_database()  # start with a clean store for the demo

class Rag:
    def __init__(self, retriever, llm):
        self.retriever = retriever
        self.llm = llm

    @instrument
    def retrieve(self, question: str) -> list[str]:
        return self.retriever.search(question, k=4)

    @instrument
    def answer(self, question: str) -> str:
        chunks = self.retrieve(question)
        prompt = "Answer using only this context:\n" + "\n".join(chunks)
        prompt += f"\n\nQuestion: {question}"
        return self.llm.complete(prompt)

app = Rag(retriever=my_retriever, llm=my_llm)
judge = JudgeProvider()

ctx = Select.RecordCalls.retrieve.rets[:]

f_groundedness = Feedback(judge.groundedness_measure_with_cot_reasons,
                          name="Groundedness").on(ctx.collect()).on_output()
f_answer_rel = Feedback(judge.relevance, name="Answer relevance") \
    .on_input().on_output()
f_context_rel = Feedback(judge.context_relevance, name="Context relevance") \
    .on_input().on(ctx).aggregate(lambda xs: sum(xs) / len(xs))

tru_app = TruApp(app, app_name="support-bot", app_version="v1",
                 feedbacks=[f_groundedness, f_answer_rel, f_context_rel])

questions = [
    "How do I reset my device?",
    "What is the warranty period?",
]
with tru_app as recording:
    for q in questions:
        app.answer(q)

print(session.get_leaderboard())

Reading the result

Suppose the leaderboard shows these averages for two versions of the same app.

These numbers are made up to show how to read a table. Here v2 improves context relevance a lot, and groundedness rises with it. Answer relevance barely changes. The lesson is that the retriever was the bottleneck. The language model was already answering the question; it just needed better evidence.

Following up on low scores

After the leaderboard, open the lowest scoring records. For each one, ask three questions.

1. Was the retrieved context missing the needed fact? That is a retrieval problem.
2. Was the fact present but ignored? That is a prompt or model problem.
3. Was the judge wrong? That is an evaluation problem, and the judge prompt may need work.

Adding a custom feedback function

You can write your own function in plain Python. For example, you can check that an answer contains a citation marker, or that a JSON output has the required keys.

import json

def has_required_keys(output: str) -> float:
    try:
        data = json.loads(output)
    except ValueError:
        return 0.0
    needed = {"order_id", "status"}
    return 1.0 if needed.issubset(data) else 0.0

f_format = Feedback(has_required_keys, name="Valid JSON keys").on_output()

Such rule-based checks are cheap, fast and fully repeatable. They make a good partner for judge-based scores.

Tracing and span structure

Modern tracing breaks one run into nested steps called spans. Understanding the shape of a trace makes TruLens records easier to read.

• Root span. The whole request, from user input to final answer. Its input and output are what users see.
• Retrieval span. One call to a retriever. It holds the query sent to the index and the list of passages returned, often with their scores.
• Generation span. One call to a language model. It holds the prompt, the model name, the settings and the output text.
• Tool span. One call to an outside function, such as a database query, a web call or a model service. It holds the arguments and the result.
• Child spans. Any of the above can contain smaller steps, such as a reranking step inside retrieval.

Feedback functions can point at any span. A groundedness score reads the generation span's output and the retrieval span's passages. A tool correctness check reads a tool span's arguments and result. Because the trace is structured, you can ask narrow questions such as "which tool call returned an empty result most often".

Comparing app versions step by step

A good comparison changes one thing at a time. The table shows typical changes and the scores that should move.

If a score outside the expected set moves, investigate before accepting the change. A prompt change that lifts groundedness but drops answer relevance may have taught the model to refuse too often.

Troubleshooting guide

Working with agents

An agent decides its own steps, so two runs on the same input can follow different paths. This makes evaluation harder, and traces more valuable. Useful checks include:

• Plan quality. Does the first plan fit the task, or does it add needless steps?
• Tool choice. Did the agent pick the right tool for each step?
• Argument validity. Were the arguments well formed and consistent with earlier results?
• Loop detection. Did the agent repeat the same call many times?
• Final answer support. Is the final answer supported by what the tools returned?
• Cost and step count. Did the agent stay within a step and token budget?

Some of these are easy to code as rules, such as step count and repeated calls. Others, such as plan quality, need a judge. See agentic RAG and multi-agent systems for the design side.

Keeping evaluation data private

Records contain user questions, retrieved text and answers. Treat them as sensitive.

• Store the database in an environment with the same access rules as the source data.
• Remove or mask personal data before it reaches an outside judge. See presidio.
• Set a retention period and delete old records.
• Do not paste real records into tickets or chats. Share a trace identifier instead.

Strengths and limitations

Strengths

• Clear focus on RAG quality. The triad gives a small, easy to explain set of checks.
• Works on live traffic. Reference-free scores need no labels.
• Traces and scores in one place. You can see why a score is low by reading the steps.
• Version comparison. Leaderboards make A/B style tuning simple.
• Open source and extensible. You can add providers and custom feedback functions.
• Integrates with common frameworks. There are wrappers for popular orchestration libraries such as LlamaIndex and LangGraph, and a generic path for custom code.

Limitations

• Judge noise. A language model judge can be inconsistent. Scores vary with the prompt, the model and sometimes the run.
• Judge bias. Judges may prefer longer or more fluent answers, or favour outputs from their own model family.
• Cost and latency. Each feedback call is a model call. Scoring many records adds cost and time.
• Score meaning is relative. A groundedness of 0.8 is not a universal threshold. It only has meaning when compared with other versions on the same inputs.
• API churn. Names and modules have changed between major versions. Pin your version.
• Not a safety layer. It measures after the fact. It does not block bad outputs at run time. For that, see guardrails.

Ways to reduce judge noise

• Use a strong judge model, and keep it fixed across comparisons.
• Ask the judge for a reason before the score.
• Set temperature low.
• Calibrate against a small set of human-labelled examples.
• Average over several judge calls for borderline cases.

How it compares with alternatives

How to choose

• If your main question is "are my RAG answers grounded", TruLens, RAGAS and DeepEval all cover it. Pick the one that fits your workflow.
• If you want evaluation to run in your normal test suite and fail a build, a test-style tool such as DeepEval or Promptfoo is natural.
• If you want to browse traces from live traffic with scores attached, a tracing platform such as Langfuse or Phoenix may feel more natural, and TruLens also fits here.
• Many teams combine tools. The metrics are not exclusive.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types such as numbers, categories, dates and text, missing values, and often no useful column names. Such a model is pre-trained on many tables, often synthetic ones, and then predicts on a new table through in-context learning: you give it labelled rows as context and it predicts the rest without gradient updates. See TabPFN, TabICL and TabSTAR for examples, and the pages on architecture and benchmarks for how such a model is built and measured.

TruLens does not train or run these models. Its role is on the application side, where a tabular model is one component of a larger LLM-driven system. Here are the concrete places where it helps.

Evaluating language model front ends over tables

Many products put a language model in front of tabular data. A user asks a question in plain words. The system picks a table, builds a query or calls a prediction model, and explains the result. TruLens can trace each of those steps and score them.

• Groundedness checks that the explanation matches the numbers returned by the table query or prediction. If the model says "revenue rose 12 percent" but the result shows 8, groundedness should drop.
• Answer relevance checks that the reply addresses the question.
• Custom feedback functions can compare numbers in the answer with numbers in the tool result, which is a more exact check than a judge model.

Tracing tool calls to tabular models

An agent may call a tabular foundation model as a tool for classification, regression, imputation or anomaly detection. Each call is a step in a trace. TruLens records the table slice sent in, the call arguments and the output. That makes it possible to answer questions such as "which rows did the agent pass as context" and "did the agent ask for a prediction on a column it should not have used". See agentic RAG and tracing.

Checking schema and column handling

Because many tables have unclear or missing column names, an LLM layer is often asked to infer meaning from values. A feedback function can test whether the layer picked the right column for a user request, or whether it described a column type correctly. This is a way to test semantic or sector classification of columns and tables, such as deciding that a column of codes is a country field.

Imputation and anomaly use cases

For imputation, an LLM front end may explain which cells were filled and why. For anomaly detection, it may explain why a row was flagged. In both cases the explanation should match the actual model output. TruLens style checks can score that match and catch invented reasons. See imputation networks and anomaly detection.

Continual learning and regression tracking

When a tabular model is updated over time, you want to know that old behaviour did not break. Treat each model release as an app version. Run a fixed set of questions and tool calls, and compare scores across versions. This is a light way to catch regressions in the LLM layer around the model. For the model itself, see continual learning work and ideas such as experience replay.

What it cannot do here

TruLens cannot measure the accuracy of a tabular model's predictions. That needs labelled holdout data and standard metrics. It also cannot judge whether pre-training data was good. Use it for the language and tool layer, and use ordinary tabular evaluation for the model.

Practical guidance and best practices

Start small

• Pick 20 to 50 representative inputs. Include easy cases, hard cases and known failures.
• Run them under the recorder and read the lowest scoring records by hand.
• Only then scale to larger sets or live traffic.

Keep the judge stable

• Fix the judge model and prompt while comparing app versions.
• If you change the judge, re-run the old versions too.
• Record the judge setup next to the results.

Combine judge scores with rules

• Use rule-based checks for anything that can be checked exactly: JSON shape, citation presence, number matching, banned phrases.
• Use judge scores for fuzzy qualities: relevance, tone, support.

Version everything

• Give each app configuration a clear version name.
• Change one thing at a time so score changes have a clear cause.
• Keep the test inputs fixed within a comparison.

Use thresholds with care

• Set a threshold from a baseline, not from a guess. For example, "no more than a small drop from the last release".
• Treat thresholds as alarms for review, not as proof.

Sample live traffic

• Scoring every live request can be costly. Score a sample, plus every request that a cheap rule flags.
• Log user feedback and check that it correlates with your scores.

Protect data

• Records can hold sensitive text. Control who can open the dashboard and where the database lives.
• Remove or mask personal data before sending text to a hosted judge. See Presidio.

Calibrate with humans

• Label a small sample by hand and compare it with judge scores.
• If they disagree often, revise the judge prompt or switch the judge model.

A simple release checklist

1. Run the fixed test set on the new version.
2. Compare the three triad scores with the last release.
3. Read the ten worst records.
4. Check cost and latency.
5. Ship only if scores hold and no new failure type appears.

A short adoption plan

A small team can adopt TruLens in a week with the steps below.

1. Day one: wrap the existing app and record 20 real questions. Open the dashboard and read five traces end to end.
2. Day two: add the three core feedback functions and run them on the 20 records. Read the lowest scores.
3. Day three: build a test set of 100 questions from logs and documents, with a few reference answers for a core subset.
4. Day four: compare the current app with one change, such as a reranker or a stricter prompt, on the same 100 questions.
5. Day five: calibrate the judge on 50 samples labelled by people, and set minimum scores for releases.

After the first week, keep the loop short. Every failure found in production becomes a test case, every change is compared on the same set, and every release records its scores next to its version.

Keep this plan flexible, and adjust the order to fit your release schedule.

Common mistakes

• Trusting a single number. An average can hide a group of bad cases. Look at the distribution and the worst records.
• Changing many things at once. If the retriever, prompt and model all change, you cannot tell which helped.
• Judging with the same model that generated. A model may rate its own style higher. Use a different judge when you can.
• Skipping human review. Judge scores are a guide. They do not replace reading examples.
• Ignoring retrieval. Many teams tune prompts when low scores come from retrieval. Check context relevance first.
• Using tiny test sets. Ten examples give noisy comparisons.
• Leaking test cases. If you tune prompts on the exact test set, scores will look better than real performance.
• Not pinning versions. The library API and the judge model can change under you.
• Scoring without traces. A score with no trace is hard to act on. Keep both.
• Using it as a guardrail. It evaluates; it does not stop bad output. Use a guardrail tool for blocking.

Frequently asked questions

What is TruLens used for?

TruLens is used to record and score the behaviour of LLM applications, especially retrieval-augmented generation apps and agents. It captures traces of each run and applies feedback functions that rate things like groundedness, context relevance and answer relevance, so you can find weak spots and compare versions of your app.

What is the RAG triad?

The RAG triad is a set of three checks for a retrieval pipeline: context relevance, groundedness and answer relevance. Context relevance asks if the retrieved passages fit the question. Groundedness asks if the answer is supported by those passages. Answer relevance asks if the answer addresses the question. Together they locate where a pipeline fails.

What is a feedback function?

A feedback function is a small program that takes parts of a recorded run and returns a score, usually between zero and one. Some call a language model as a judge, some use embeddings or classifiers, and some are plain code. You choose which parts of the record each function reads, then TruLens stores the scores with the record.

Does TruLens need labelled data?

No, not for its core RAG checks. Groundedness, context relevance and answer relevance are reference-free, so they work on live traffic with no ground truth. If you also want to measure agreement with known correct answers, you can add reference-based feedback functions and supply a labelled test set.

How reliable are the scores?

They are useful but noisy. A judge model can vary between runs and can have biases toward fluent or long answers. Reliability improves with a strong, fixed judge, low temperature, reasons before scores and checks against a small human-labelled sample. Treat scores as comparisons between versions, not as absolute truth.

How is TruLens different from DeepEval?

Both score LLM outputs. TruLens leans toward tracing a run and attaching feedback functions with a dashboard for comparing app versions. DeepEval leans toward unit-test style metrics that run inside a normal test framework and can fail a build. Many teams use both for different stages of work.

Can I use it with local or open models?

Yes. The judge and the app can be any model your setup can call, as long as a provider exists or you write one. Local judges can reduce cost and keep data in house, though smaller judges may be less consistent, so check them against human labels first.

Can TruLens evaluate agents and not only RAG?

Yes. It records traces of steps and tool calls, so you can score parts of an agent run, such as whether a tool call was sensible or whether the final answer matches the tool result. You may need custom feedback functions for agent-specific checks.

Does it help with tabular foundation models?

Indirectly. It does not measure prediction accuracy of a tabular model. It helps evaluate the language and tool layer around such a model, for example whether an explanation matches the numbers returned, or whether an agent called the model with the right columns and rows.

How do I avoid high evaluation cost?

Score a sample of live traffic, use rule-based checks where possible, run feedback in batches off the request path, cache repeated judge calls, and use a smaller judge for easy checks while keeping a stronger one for hard cases. Always confirm that the cheaper setup still agrees with human labels.

Glossary

• Agent: A program in which a language model chooses steps and calls tools.
• Answer relevance: A score for whether the answer addresses the question.
• Chunk: A passage of a document stored and retrieved as one unit.
• Context relevance: A score for whether retrieved passages match the question.
• Embedding: A list of numbers that represents the meaning of a text.
• Feedback function: A function that scores part of a run.
• Groundedness: A score for whether claims are supported by the retrieved context.
• Hallucination: A fluent statement that is not supported by facts or sources.
• In-context learning: Learning from examples placed in the input, without weight updates.
• Judge model: A language model used to score outputs.
• Leaderboard: A ranked view of app versions by score.
• Observability: The ability to understand a running system from its logs, metrics and traces.
• RAG: Retrieval-augmented generation, where retrieved text is given to the model as context.
• Record: The stored data of one run.
• Reference-free: A score that needs no known correct answer.
• Regression: A drop in quality after a change.
• Selector: A pointer to part of a record.
• Trace: The ordered steps inside a run.

Summary

TruLens records what an LLM app does and scores it with feedback functions. Its best known use is the RAG triad of context relevance, groundedness and answer relevance, which helps you tell whether a failure comes from retrieval, from generation or from the question itself. Its dashboard lets you compare app versions on the same inputs.

Use it with care. Judge scores are noisy, so fix the judge, combine it with exact rule checks and calibrate against human labels. For tabular foundation models, it works at the application layer: tracing tool calls to the model, checking that explanations match the numbers, and tracking regressions across releases. Model accuracy itself still needs standard tabular evaluation. For more on how such models are built and measured, see research, architecture and benchmarks.

Related

It belongs to RAG in my stack, under LLMs, retrieval, and evaluation. Others in the same group: