RAGAS

Overview

RAGAS is an open-source Python library for evaluating retrieval-augmented generation (RAG) systems. A RAG system retrieves text from a knowledge base and gives it to a language model, which writes an answer. RAGAS measures how well each part of that chain works. It scores whether the answer is supported by the retrieved text (faithfulness), whether the answer addresses the question (answer relevancy), whether the retrieved text is ranked well and is on topic (context precision), and whether the retrieved text covers what was needed (context recall).

The library matters because RAG systems fail in quiet ways. An answer can sound confident and still be wrong, and the cause may lie in retrieval, in the prompt or in the model. RAGAS gives numbers for each part, so you can see where to act. Many of its metrics use a language model as a judge and need no hand-written reference answers for some checks, which makes it practical for fast iteration. It is often used with TruLens and DeepEval, and it pairs well with retrieval components like BM25, ColBERT and cross-encoder reranking.

What it is

RAGAS is a toolkit with three main jobs.

• Metrics. A set of scoring functions for the parts of a RAG pipeline: retrieval, generation and the match between them.
• Test set generation. Helpers that create question and answer examples from your own documents, so you can start evaluating without a hand-built benchmark.
• Integration. Hooks that let you run the metrics over data from common orchestration and tracing tools and log the results.

The name RAGAS is usually read as short for "Retrieval Augmented Generation Assessment". Details of the API change between releases, so use the documentation of the version you install.

What a RAG sample looks like

An evaluation sample usually has these fields.

Some metrics need only the first three. Others, such as context recall, need the reference, since they compare what was retrieved with what was needed.

An everyday analogy

Imagine a student writing an essay with a stack of library books. Several questions arise. Did the student pick the right books? Are the best books on top of the pile? Did the student cover everything the books say that matters? Did the student write only what the books support, or did they invent facts? Did the essay answer the question that was set? RAGAS plays the role of a teacher who checks each of these in turn, and gives a score for each, so that you can tell if the problem is the book selection or the writing.

What it is not

RAGAS is not a retriever, a vector store or a generator. It does not improve answers by itself. It also does not replace human review. Its judge-based scores are estimates, and they should be calibrated against human labels for any high-stakes use.

Where it came from

RAGAS was introduced in 2023 in a paper titled "RAGAS: Automated Evaluation of Retrieval Augmented Generation". The paper arrived when RAG was becoming the standard way to ground language models in private or fresh data, and teams needed a way to evaluate such systems without large human-labelled sets. The authors proposed reference-free metrics that use a language model to check parts of the pipeline, and an open-source library to apply them.

Earlier evaluation work for retrieval relied on ranking metrics such as recall at k and nDCG, which need labelled relevant documents. Evaluation of text generation relied on overlap metrics such as BLEU and ROUGE, which compare an answer with a reference by word overlap and often fail to capture meaning or factual support. The new idea was to break a RAG answer into claims, check each claim against the retrieved context, and break a question into what the context needed to cover, using language models for these steps. This follows the broader trend of "LLM as judge" evaluation. See LLM-as-judge.

The library has since grown. New metrics were added, including ones for noise sensitivity, entity coverage and agent behaviour, and tools for generating test sets. Names of metrics and parameters have also been revised more than once, so always check the current documentation. For context on the wider research on retrieval and evaluation, see research.

How it works

Step 1: Collect evaluation samples

You gather a set of questions. For each, you run your RAG system and record the retrieved passages and the final answer. If you have reference answers, you add them. You can also generate samples from your documents with the library's test set generator.

Step 2: Choose metrics

Pick metrics for the question you want to answer.

• To test retrieval, use context precision and context recall.
• To test grounding, use faithfulness.
• To test whether the answer addresses the question, use answer relevancy.
• To test overall correctness against a reference, use answer correctness or a similar metric.

Step 3: Judge with a language model

Most metrics call a language model several times. The library breaks a metric into smaller tasks, asks the model to do each, and combines the results into a score.

Step 4: Aggregate and inspect

The output is a score per sample per metric, and averages across the set. You inspect the low-scoring samples to find patterns, change the pipeline, and run again.

The core metrics in detail

Faithfulness

Does the answer stick to the retrieved context? The method has two steps. First, the judge splits the answer into separate statements, called claims. Second, for each claim, the judge checks whether the retrieved context supports it. The score is the share of claims that are supported.

faithfulness = supported claims / total claims

A low score means the model said things the context did not support, which is a sign of hallucination or of using knowledge from outside the context. This metric does not need a reference answer.

Answer relevancy

Does the answer address the question? One common method asks the judge to write several questions that the answer would be a good reply to, then compares those generated questions with the original using embedding similarity. If the answer is on topic, the generated questions look like the original. If it rambles or dodges, they differ. The score is the average similarity. This measures relevance, not truth. An answer can be relevant and wrong.

Context precision

Are the useful passages ranked high? For each retrieved passage, the judge decides whether it is useful for answering the question. The metric then rewards lists where useful passages come first. A list with relevant passages at ranks 1 and 2 scores better than one with them at ranks 4 and 5, even if both have the same passages. This tells you about retriever and reranker quality. See cross-encoder reranking.

Context recall

Did retrieval find everything needed? Using a reference answer, the judge splits it into statements and checks whether each could be found in the retrieved context. The score is the share of reference statements that are covered.

context recall = reference statements found in context / total reference statements

A low score means the retriever missed key information, so even a perfect generator could not answer fully. This metric needs a reference.

Other metrics

The library includes more metrics, which vary by version. Examples include:

• Context entity recall: whether important entities from the reference appear in the context.
• Noise sensitivity: how much irrelevant context harms the answer.
• Answer correctness and semantic similarity: how close the answer is to a reference by facts and by meaning.
• Aspect critique style checks: custom yes or no judgments such as harmlessness or conciseness.
• Agent and tool use metrics: whether an agent followed the expected steps.

Test set generation

To avoid writing hundreds of questions by hand, the library can build a synthetic test set. A typical approach loads your documents, extracts key information and builds a graph of related chunks. It then asks a language model to write questions of different types, such as simple fact lookups, questions that need information from several chunks, and more abstract questions. Each question comes with a reference answer and the source passages.

Synthetic test sets are a fast start. They are not a replacement for real user questions. Mix both, and review a sample by hand. See also synthetic data generation.

Where the judge comes from

Metrics call a language model that you choose. This can be a hosted model or a local one. The embedding model for answer relevancy is also configurable. Because the judge is a model, scores depend on which judge you use, its version and its prompt. Fix them for a given comparison.

Diagnosing a pipeline with the four metrics

The four core metrics form a simple decision table. Read the pattern, not a single number.

This table is a starting guide. Real failures mix causes, so confirm each diagnosis by reading a few samples before making a large change.

Writing a custom metric

Sometimes the built-in metrics do not match your rules. A support team may require that every answer cites a policy section. A finance team may require that every number in an answer appears in the context. You can write a custom check in two ways.

• Rule-based. Use code, such as a regular expression for citations or a function that extracts numbers from the answer and the context and compares them. Rule-based checks are cheap, exact and repeatable.
• Judge-based. Write a short instruction for the judge model with a clear yes or no question, and ask for a one-line reason. Keep the question narrow. "Does the answer give a refund amount that appears in the context?" is better than "Is the answer good?"

import re

def numbers_grounded(answer: str, contexts: list[str]) -> float:
    nums = re.findall(r"\d+(?:\.\d+)?", answer)
    if not nums:
        return 1.0
    joined = " ".join(contexts)
    found = [n for n in nums if n in joined]
    return len(found) / len(nums)

A rule like this is not a replacement for a judge, but it is a good guard for numbers, which language judges often check poorly.

Sampling live traffic

Offline test sets show how the system behaves on questions you know about. Live traffic shows what users actually ask. A light monitoring loop looks like this.

1. Log each request with its question, retrieved passages, answer and a version tag.
2. Sample a small share of requests each day, with extra weight on requests that got a thumbs down or a follow-up complaint.
3. Score the sample with a few metrics, mostly faithfulness and context precision.
4. Plot the scores by day and by version.
5. Send the lowest-scoring cases for human review, and add confirmed failures to the offline test set.

This loop keeps the test set fresh. It also catches drift, for example when a new batch of documents lowers retrieval quality.

Handling multi-turn conversations

Real assistants carry a conversation. A follow-up such as "and what about the second one?" is meaningless without history. Evaluate in two parts.

• Check that the question was rewritten into a standalone form before retrieval. See query rewriting.
• Score the rewritten question's retrieval with the usual context metrics.

If you score the raw follow-up directly, retrieval metrics will look bad for reasons that are not a retrieval problem.

Reporting results to a team

A short report is more useful than a dashboard nobody reads. Include these items.

• The pipeline version, the test set version, the judge model and the library version.
• The four core metrics with averages and the share of samples below a chosen threshold.
• The three most common failure patterns, each with two example samples.
• The change since the last release, and whether it is larger than the run-to-run noise.
• One recommended action.

Keep the report short and repeat the same layout each time, so that readers can compare releases at a glance.

Key concepts and terms

A worked example

A numeric walk-through

Suppose a user asks: "What is the notice period to cancel the plan, and is it refundable?" The retriever returns three passages, and the model writes an answer.

Faithfulness. The judge splits the answer into three claims. Claim 1 (30 days notice) is supported by context 1. Claim 2 (no refund of current month fees) is supported by context 2. Claim 3 (a free month after cancelling) has no support. The score is 2 / 3, about 0.67. The low score points to an invented claim.

Context recall. The reference has two statements. Both can be found in the contexts. The score is 2 / 2, which equals 1.0. Retrieval found what it needed.

Context precision. Contexts 1 and 2 are useful, and context 3 is not. Because the useful ones are ranked first, the score is high, close to 1.0. If context 3 had been ranked first, the score would drop.

Answer relevancy. The answer addresses notice period and refund, so it scores high. The extra invented line does not change relevance much.

The diagnosis is clear: retrieval worked, but generation added an unsupported claim. The fix is on the generation side, for example a stricter prompt such as "answer only from the context", or a different model. A single overall score would have hidden this.

Python example

The code below shows the general shape. Function and class names differ between releases, so check the current documentation.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

data = {
    "user_input": ["What is the notice period to cancel the plan, and is it refundable?"],
    "retrieved_contexts": [[
        "Cancellation requires 30 days written notice.",
        "Fees paid for the current month are not refunded.",
        "Support is available on weekdays.",
    ]],
    "response": ["You must give 30 days written notice. Fees for the current month "
                 "are not refunded. You also get a free month after cancelling."],
    "reference": ["30 days written notice is required. Fees for the current month are not refunded."],
}

dataset = Dataset.from_dict(data)
result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    # llm=..., embeddings=...   # configure your judge and embedding model here
)
print(result)

The judge model and the embedding model are set through configuration. Use a placeholder endpoint such as example.internal for a local gateway, and keep keys in your secret store, never in the code.

Comparing two pipeline versions

versions = {"v1_bm25_only": run_pipeline_v1, "v2_hybrid_rerank": run_pipeline_v2}
rows = []
for name, pipeline in versions.items():
    ds = build_dataset(questions, pipeline)     # runs the pipeline and records contexts and answers
    scores = evaluate(ds, metrics=[faithfulness, context_precision, context_recall])
    rows.append({"version": name, **scores})

A table of version against metric shows what each change did. For example, adding a reranker should raise context precision. Switching to a stricter prompt should raise faithfulness. If a change moves a different metric than you expected, investigate before you trust it.

What a good workflow looks like

1. Build 50 to 200 questions from real user logs and from generated samples.
2. Label a small subset by hand to calibrate the judge.
3. Run the baseline pipeline and save the scores.
4. Change one thing at a time, run again, and compare.
5. Read the lowest-scoring samples for each metric.
6. Add every real failure you meet to the test set so that it stays fixed.

Strengths and limitations

Strengths

• Component-level insight. Separate scores for retrieval and generation help locate the cause of a failure.
• Fast start. Reference-free metrics and test set generation let you begin without a large labelled set.
• Cheaper than human review. You can score thousands of samples in an hour for modest cost.
• Simple concepts. The four main metrics are easy to explain to non-specialists.
• Open source and extensible. You can add custom metrics and swap the judge model.
• Fits regression testing. Run it on each release to catch quality drops. See regression boards.

Limitations

• Judge noise. Scores come from a language model, so they vary with the model, the prompt and sometimes between runs.
• Judge bias. Judges may prefer longer or more fluent answers, and may miss subtle errors.
• Cost and latency. Each metric makes several model calls per sample, so large sets can be costly.
• Reference quality. Context recall and correctness depend on reference answers. Poor references give poor scores.
• Not a proof of truth. Faithfulness checks support from the retrieved context, not whether the context itself is correct.
• Version drift. Metric definitions and prompts change between releases, so scores may not be comparable across versions.
• Synthetic set limits. Generated questions can be easier or more uniform than real user questions.
• Language and domain gaps. Judges work best in languages and fields they know well.

How it compares with alternatives

Choosing between them

• Use RAGAS when you want clear, RAG-specific metrics and quick test set creation.
• Use TruLens, Phoenix, LangSmith or Langfuse when you also need traces and a dashboard.
• Use DeepEval or Promptfoo when you want evaluation to look like unit tests in a build pipeline.
• Many teams combine two: one for traces and one for metrics. They are not exclusive.

Related practices

Evaluation is more reliable when judge scores are checked against people, and when the judge setup is stable. See LLM-as-judge and evals for general guidance. Trace data from OpenTelemetry and tracing tools can supply the retrieved contexts and answers that RAGAS scores.

Role in tabular foundation models

RAGAS evaluates RAG systems, which work on text. A tabular foundation model works on tables. The two meet when a product wraps a table model with a language interface, or uses retrieval around a table model. In those cases, RAGAS-style checks apply to the language parts, and standard tabular evaluation applies to the numeric parts.

Background on tabular foundation models

A tabular foundation model is pre-trained on many tables, often synthetic, so it can predict on a new table without task-specific training. It handles rows and columns, mixed types, missing values and unnamed columns, and it learns from a few labelled rows given as context. Examples include TabPFN, TabICL and TabSTAR. The architecture page explains how such a model is built.

Where RAGAS-style evaluation helps

• Natural-language assistants over tables. A user asks, "Which customers are most likely to churn, and why?" A pipeline calls a tabular model, gets predictions and feature effects, and has a language model explain them. Faithfulness can check whether the explanation only states what the model output supports. If the text says "region drove churn" and the output shows no such effect, the claim is unsupported.
• Retrieval of documentation. An assistant often retrieves data dictionaries, column descriptions and business rules. Context precision and recall measure whether the right definitions were found. This matters because an assistant that picks the wrong column definition gives a confident but wrong analysis.
• Retrieval of context rows. Some pipelines select similar labelled rows to feed a tabular model as in-context examples. The ideas of precision and recall carry over: are the selected rows useful, and are the needed ones included? Here you evaluate with tabular metrics, not a language judge, but the way of thinking is the same.
• Sector and semantic classification. When text fields are mapped to categories, retrieval over category descriptions can be scored for recall at k, and a language model's explanation of the choice can be scored for faithfulness.
• Imputation and anomaly explanations. When a system explains why a value was imputed or why a row was flagged, those explanations are generated text that should be grounded in the model's output and data. Faithfulness-style checks catch invented reasons. See anomaly detection and masked cell modeling.
• Text-to-SQL layers. When questions are turned into queries over tables, retrieval of schema information can be scored with context recall, and the final answer can be checked against the query result. See text-to-SQL style tools for the call pattern.

What RAGAS cannot judge

It cannot tell you whether a tabular model's predictions are accurate. For that, use standard tabular metrics such as accuracy, area under the curve, error measures and calibration, on held-out data. It also cannot verify numbers against a table unless you supply the table output as the context. The clean split is: tabular benchmarks for predictive quality, and RAGAS-style checks for the language and retrieval layer. For measured results of tabular models, see benchmarks, and for the research landscape see research.

A simple pattern for a table assistant

1. Log each interaction: the user question, the retrieved documentation, the model outputs and the final text.
2. Treat the model outputs and retrieved documentation as the retrieved contexts.
3. Score faithfulness of the final text against those contexts.
4. Score answer relevancy for the question.
5. Score context recall on a labelled set of questions with known required definitions.
6. Track tabular accuracy separately on held-out data.

Practical guidance and best practices

Prepare the data

• Use questions that look like real ones. Pull them from logs, support tickets or search queries.
• Keep a reference answer for at least a core set, so that recall and correctness can be measured.
• Remove personal data before sending samples to any external judge. See presidio.
• Version your test set and never edit it silently.

Control the judge

• Fix the judge model and version for a comparison. Do not change it halfway.
• Use a strong judge for hard metrics such as faithfulness.
• Set low randomness for scoring calls.
• Calibrate against human labels on 50 to 100 samples. Record agreement and note where the judge is weak.
• Run the same set twice to estimate noise, and treat differences smaller than that noise as unreal.

Read the numbers well

• Look at distributions, not just averages. A few very bad answers matter more than a small mean shift.
• Read low-scoring samples by hand. The score tells you where to look, and the text tells you why.
• Compare versions on the same questions.
• Track scores over time and add them to a regression board.

Design the pipeline around the metrics

• If context recall is low, improve retrieval: try hybrid search, better chunking, query rewriting and more candidates. See hybrid search, query rewriting and contextual retrieval.
• If context precision is low, rerank and trim: add a reranker, use a score threshold, pass fewer passages.
• If faithfulness is low, tighten the prompt, require citations, lower temperature or add a verification step. See guardrails.
• If answer relevancy is low, check that the question is clear, handle follow-up questions with history rewriting, and shorten the prompt.

Control cost

• Score a sample of live traffic, not all of it.
• Use a cheaper judge for coarse screening and a stronger one for flagged cases.
• Cache judge calls for repeated samples.
• Run evaluations in batches off the request path.

Keep it in the release process

• Run the test set on every change to the prompt, retriever, chunking or model.
• Set minimum scores that block a release when missed.
• Store results with the commit or version identifier.

Common mistakes

• Using one overall score. The value of RAGAS is in the separate metrics. Averaging them hides the cause.
• Trusting the judge blindly. Without calibration, scores can drift from human opinion.
• Changing the judge between runs. Comparisons then mean nothing.
• Poor references. If the reference answer is wrong or incomplete, recall and correctness are wrong too.
• Testing only synthetic questions. They are useful but often too clean. Add real questions.
• Too few samples. A set of ten questions gives noisy results.
• Confusing faithfulness with truth. A faithful answer can repeat a false statement from the context.
• Ignoring cost. Running every metric on every request in production adds up quickly.
• Leaking test questions into prompts or fine-tuning. This inflates scores.
• Skipping the human look. Metrics point to problems. They do not replace reading answers.
• Applying text metrics to numbers. A language judge is weak at checking arithmetic over large tables. Verify numbers by code, not by a judge.
• Not pinning library versions. Metric definitions change, so pin the version and note it in reports.

Mistakes in judge usage

• Asking the judge vague questions. Narrow yes or no questions give steadier scores than broad ratings.
• Letting the judge see the answer's source model name. This can introduce bias in some setups. Remove such labels.
• Using the same model as generator and judge without checking. A model may rate its own style higher. Compare with a different judge on a sample.
• Treating a single run as final. Judge noise means one run can mislead. Repeat important comparisons.
• Ignoring position and length effects. In comparisons of two answers, shuffle the order and watch for a preference for longer text.

Frequently asked questions

What is RAGAS used for?

RAGAS is used to measure the quality of retrieval-augmented generation systems. It scores whether answers are supported by the retrieved text, whether they address the question, whether the retrieved passages are on topic and well ranked, and whether the retrieval found everything that was needed. Teams use it to compare versions and to catch regressions.

Do I need reference answers to use RAGAS?

Not for every metric. Faithfulness and answer relevancy can be computed without a reference, because they compare the answer with the context and the question. Context recall and answer correctness do need a reference. A good plan is to write references for a core set of questions and use reference-free metrics for the rest.

What does faithfulness measure?

Faithfulness measures how much of the answer is supported by the retrieved context. The answer is split into separate claims, each claim is checked against the context, and the score is the share of claims that are supported. A low score points to invented statements or reliance on knowledge from outside the context.

What is the difference between context precision and context recall?

Context precision asks whether the useful passages are ranked high in the retrieved list, so it reflects the quality of ranking and the amount of noise. Context recall asks whether the retrieved list covers the information needed to answer, so it reflects what was missed. A system can have high precision and low recall, or the reverse.

Is RAGAS accurate?

It is a useful estimate, not a perfect measurement. Because language models do the judging, scores can vary with the judge, the prompt and the run. Calibrate against human labels, fix the judge setup, and read low-scoring samples. Treat small score differences with caution and focus on clear, repeatable changes.

Which language model should I use as the judge?

Use a model that is strong in your language and domain, and keep it fixed for all comparisons. A stronger judge usually agrees better with people, especially for faithfulness. Some teams use a cheaper judge for broad screening and a stronger one for disputed cases. Always check agreement with human labels.

Can RAGAS generate test questions?

Yes. It can build question and answer examples from your documents, including simple fact questions and questions that need several passages. This helps you start quickly. Review a sample by hand, and mix in real user questions, because generated sets tend to be cleaner and easier than real traffic.

How is RAGAS different from TruLens?

RAGAS focuses on a set of RAG metrics and test set generation. TruLens focuses on instrumenting an application, recording traces and scoring them with feedback functions, with a dashboard to compare versions. They overlap in the idea of scoring groundedness and relevance, and many teams use both, one for traces and one for batch metrics.

Can I use RAGAS with a tabular foundation model?

You can use it on the language and retrieval parts of a system that wraps a tabular model, such as an assistant that explains predictions or retrieves column documentation. It does not measure the predictive accuracy of the tabular model itself. For that, use standard held-out evaluation. The two kinds of checks cover different layers.

How big should my evaluation set be?

Start with 50 to 200 questions that reflect real use, and grow it as you find new failures. Small sets are useful for fast checks but noisy. Larger sets give more stable scores, and you can stratify by question type to see where the system is weak. Always keep a stable core set for regression checks.

Glossary

• Answer relevancy: A score for whether the answer addresses the question.
• Calibration: Checking that automatic scores agree with human labels.
• Claim: A single statement taken from an answer.
• Context precision: A score for how well useful passages are ranked.
• Context recall: A score for how much needed information was retrieved.
• Embedding: A vector that represents the meaning of a text.
• Evaluation sample: One record with question, context, answer and optional reference.
• Faithfulness: The share of answer claims supported by the context.
• Feedback function: A function that scores part of an application run.
• Grounding: Basing an answer on provided sources.
• Hallucination: A fluent statement that lacks support.
• Judge model: A language model used to score outputs.
• Reference: A known good answer.
• Reference-free metric: A metric that does not need a reference.
• Regression: A drop in quality after a change.
• Retrieved context: The passages returned by the retriever.
• Synthetic test set: Questions and answers generated from documents.
• Temperature: A setting that controls randomness in model output.

Summary

RAGAS gives a RAG system a set of separate scores: faithfulness for grounding, answer relevancy for on-topic answers, context precision for ranking quality and context recall for coverage. Because the scores are separate, they help you tell whether a failure comes from retrieval or from generation. Reference-free metrics and test set generation make it quick to start.

Its scores come from language model judges, so fix the judge, calibrate against human labels and read the worst samples. For tabular foundation models, RAGAS-style checks apply to the assistant and retrieval layers around the model, such as explanations and documentation lookup, while predictive quality still needs standard tabular evaluation. For more on how such models are built and measured, see research, architecture and benchmarks.

Related

It belongs to RAG in my stack, under LLMs, retrieval, and evaluation. Others in the same group: