# Evaluating RAG: faithfulness, context precision and recall

A RAG system can fail in retrieval or in generation. These four metrics tell you which. Break things on purpose and watch which score drops.

<div data-sim="ragas"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The left side shows the question, the retrieved contexts (useful ones marked) and the answer split into claims with a verdict on each. The right side shows the four scores and what they point to.

## Things to try

- Untick the unsupported claim: faithfulness goes to 1.
- Drop the best context: recall falls, which is a retrieval problem.
- Sample 2 shows a context ranked too low: precision drops.

## How it works in real systems

RAGAS and DeepEval compute these metrics with an LLM judge; the agent guide on this site shows the code. Run them on a fixed question set on every change, and spot-check the judge against human labels.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Build an evaluation set

Each row: the question, the contexts your system retrieved, its answer, and a reference answer written by a person.

```python
rows = [{
    "user_input": "Can I get my money back if my tent arrived torn?",
    "retrieved_contexts": ["Damaged or defective items qualify for a full refund or a free replacement; send a photo of the damage to support first.",
                           "Parcels damaged in transit must be reported to the carrier within 48 hours of delivery."],
    "response": "Yes: send a photo of the damage to support for a refund or replacement, and report transit damage to the carrier within 48 hours. Refunds arrive the same day.",
    "reference": "Yes. Damaged items qualify for a full refund or replacement after you send a photo; report transit damage to the carrier within 48 hours.",
}]
```

### Step 2 · Score it with RAGAS

```python
# pip install "ragas>=0.2" langchain-openai
from ragas import evaluate, EvaluationDataset
from ragas.metrics import Faithfulness, LLMContextPrecisionWithReference, LLMContextRecall, ResponseRelevancy
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini", temperature=0))
result = evaluate(dataset=EvaluationDataset.from_list(rows),
                  metrics=[Faithfulness(), LLMContextPrecisionWithReference(), LLMContextRecall(), ResponseRelevancy()],
                  llm=llm, embeddings=LangchainEmbeddingsWrapper(OpenAIEmbeddings()))
print(result.to_pandas().iloc[:, -4:])
```

The invented "same day" sentence lowers faithfulness: a generation problem, not a retrieval one.

### Step 3 · The same checks as unit tests with DeepEval

```python
# pip install deepeval
from deepeval import evaluate as de_eval
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
case = LLMTestCase(input=rows[0]["user_input"], actual_output=rows[0]["response"], retrieval_context=rows[0]["retrieved_contexts"])
de_eval(test_cases=[case], metrics=[FaithfulnessMetric(threshold=0.8), AnswerRelevancyMetric(threshold=0.7)])
```

Run the set on every change (prompt, chunking, model) and in CI; a drop in a metric points to the stage that regressed.

<div class="cl-end"></div>

## More simulations

[← HNSW search](/ai-simulations/hnsw-vector-search) · [All AI Simulations](/ai-simulations) · [LangGraph state →](/ai-simulations/langgraph-state-checkpoints)
