Özer Cem KELAHMETKELAHMET
AI Simulations · Retrieval (RAG)

Evaluating RAG: faithfulness, context precision and recall

A RAG system can fail in retrieval or in generation. These four metrics tell you which. Break things on purpose and watch which score drops.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

The left side shows the question, the retrieved contexts (useful ones marked) and the answer split into claims with a verdict on each. The right side shows the four scores and what they point to.

Things to try

  • Untick the unsupported claim: faithfulness goes to 1.
  • Drop the best context: recall falls, which is a retrieval problem.
  • Sample 2 shows a context ranked too low: precision drops.

How it works in real systems

RAGAS and DeepEval compute these metrics with an LLM judge; the agent guide on this site shows the code. Run them on a fixed question set on every change, and spot-check the judge against human labels.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · Build an evaluation set

Each row: the question, the contexts your system retrieved, its answer, and a reference answer written by a person.

rows = [{
    "user_input": "Can I get my money back if my tent arrived torn?",
    "retrieved_contexts": ["Damaged or defective items qualify for a full refund or a free replacement; send a photo of the damage to support first.",
                           "Parcels damaged in transit must be reported to the carrier within 48 hours of delivery."],
    "response": "Yes: send a photo of the damage to support for a refund or replacement, and report transit damage to the carrier within 48 hours. Refunds arrive the same day.",
    "reference": "Yes. Damaged items qualify for a full refund or replacement after you send a photo; report transit damage to the carrier within 48 hours.",
}]

Step 2 · Score it with RAGAS

# pip install "ragas>=0.2" langchain-openai
from ragas import evaluate, EvaluationDataset
from ragas.metrics import Faithfulness, LLMContextPrecisionWithReference, LLMContextRecall, ResponseRelevancy
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini", temperature=0))
result = evaluate(dataset=EvaluationDataset.from_list(rows),
                  metrics=[Faithfulness(), LLMContextPrecisionWithReference(), LLMContextRecall(), ResponseRelevancy()],
                  llm=llm, embeddings=LangchainEmbeddingsWrapper(OpenAIEmbeddings()))
print(result.to_pandas().iloc[:, -4:])

The invented "same day" sentence lowers faithfulness: a generation problem, not a retrieval one.

Step 3 · The same checks as unit tests with DeepEval

# pip install deepeval
from deepeval import evaluate as de_eval
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
case = LLMTestCase(input=rows[0]["user_input"], actual_output=rows[0]["response"], retrieval_context=rows[0]["retrieved_contexts"])
de_eval(test_cases=[case], metrics=[FaithfulnessMetric(threshold=0.8), AnswerRelevancyMetric(threshold=0.7)])

Run the set on every change (prompt, chunking, model) and in CI; a drop in a metric points to the stage that regressed.

More simulations

← HNSW search · All AI Simulations · LangGraph state →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open