Building an agentic AI system from scratch with LangGraph, Langfuse, DeepEval and RAGAS
An agent is a loop: a model decides, a tool acts, the result goes back to the model. This guide builds that loop with LangGraph, adds tracing with Langfuse, unit tests with DeepEval and retrieval scores with RAGAS, and shows a run step by step in a simulation you can play with.
- Reading time
- 13 min
- Sections
- 11
- Code examples
- 13
- Simulations
- 1
- Sources
- 6
- Published
- 17 Nov 2026
Most "agent" demos hide the parts that make an agent usable in production: a clear state, a loop that stops, a trace of every step, and tests that fail when quality drops. This guide builds all four from an empty folder.
The example is a support agent for a small online shop. It answers questions with two tools: a calculator for exact numbers and a search over the shop's policy documents. The stack has one job per tool:
- LangGraph runs the loop: which step comes next, what the state holds, when to stop.
- Langfuse records every model call and tool call as a trace, with timings, tokens and cost.
- DeepEval turns quality into unit tests that run in CI, like pytest for answers.
- RAGAS scores the retrieval half: did the right documents reach the model, and did the answer stay faithful to them.
Think of a kitchen. LangGraph is the order of work on the ticket, Langfuse is the camera over the pass, DeepEval is the tasting before a dish leaves, and RAGAS checks that the cook used the right ingredients from the pantry.
The simulation: one run, step by step
Pick a question and press Run. The left side is the graph: the dot moves from agent to tools and back until the model decides it can answer. The right side shows what the three other tools see: the growing message state, the trace as a waterfall of spans, and the evaluation scores at the end.
Interactive agent simulation. It needs JavaScript; the steps are also described in the text below.
The run is scripted so it behaves the same every time. Timings and token counts are typical values for a small model, not measurements. The third question shows the most important pattern in practice: the first search returns the wrong documents, a grading step notices, and the agent searches again with a better query.
Step 1: the project and the model
Start with a clean environment and pin versions. Agent libraries move quickly, and an unpinned upgrade is the most common reason a working agent stops working.
python -m venv .venv && source .venv/bin/activate
pip install "langgraph" "langchain-openai" "langfuse" "deepeval" "ragas" "pytest"
pip freeze > requirements.lock
Any chat model with tool calling works. The examples use an OpenAI-compatible client because many providers and local servers (vLLM, llama-server, Ollama) speak that protocol; change base_url and model to point at your own.
# app/model.py
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0) # or base_url="http://localhost:8000/v1", model="your-model"
Temperature zero does not make a model deterministic, but it removes the largest source of run-to-run variation, which matters when you test.
Step 2: tools are plain functions with clear contracts
A tool is a Python function with a name, typed arguments and a docstring. The model reads the docstring to decide when to call it, so write it for the model: what the tool does, what it returns, when not to use it.
# app/tools.py
from langchain_core.tools import tool
import ast, operator as op
_OPS = {ast.Add: op.add, ast.Sub: op.sub, ast.Mult: op.mul, ast.Div: op.truediv, ast.Pow: op.pow, ast.USub: op.neg}
def _eval(node):
if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)):
return node.value
if isinstance(node, ast.BinOp) and type(node.op) in _OPS:
return _OPS[type(node.op)](_eval(node.left), _eval(node.right))
if isinstance(node, ast.UnaryOp) and type(node.op) in _OPS:
return _OPS[type(node.op)](_eval(node.operand))
raise ValueError("only numbers and + - * / ** are allowed")
@tool
def calculator(expression: str) -> str:
"""Evaluate an arithmetic expression exactly, e.g. '0.17*2340'. Use it for any calculation."""
return str(_eval(ast.parse(expression, mode="eval").body))
@tool
def search_docs(query: str) -> str:
"""Search the shop's policy documents. Returns up to 3 passages, each starting with its source id."""
hits = retriever.invoke(query) # defined in step 3
return "\n\n".join(f"[{d.metadata['source']}] {d.page_content}" for d in hits)
Two rules keep tools safe. Never pass model output to eval or a shell: the calculator above parses the expression and allows only arithmetic. And return text the model can quote, with a source id, so answers can cite where a fact came from.
Step 3: a small retriever
Retrieval quality decides most of an agent's answer quality, so start simple and measure it before tuning anything. Split the documents into passages, embed them, keep them in a vector store and return the top three.
# app/retriever.py
from langchain_core.documents import Document
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings
docs = [
Document("Damaged or defective items qualify for a full refund or replacement; send a photo first.",
metadata={"source": "refund-policy §2"}),
Document("Refurbished laptops are covered for 12 months; batteries for 6 months.",
metadata={"source": "refurbished §3"}),
# ... the rest of your passages, 200 to 500 words each
]
store = InMemoryVectorStore.from_documents(docs, OpenAIEmbeddings(model="text-embedding-3-small"))
retriever = store.as_retriever(search_kwargs={"k": 3})
In production the in-memory store becomes pgvector, Qdrant or another vector database, and the plain vector search often becomes hybrid search with a reranker. The RAG types guide explains when each upgrade pays.
Step 4: the graph
LangGraph models the agent as a state machine. The state is a typed dictionary; here it is the list of messages. Nodes are functions that read the state and return an update. Edges say which node runs next; a conditional edge decides at run time.
# app/graph.py
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode, tools_condition
from langgraph.checkpoint.memory import MemorySaver
from langchain_core.messages import SystemMessage
from app.model import llm
from app.tools import calculator, search_docs
TOOLS = [calculator, search_docs]
SYSTEM = SystemMessage(
"You are the shop's support assistant. Use search_docs for any policy question and cite the source id "
"in square brackets. Use calculator for every calculation. If the documents do not answer, say so.")
model = llm.bind_tools(TOOLS)
def agent(state: MessagesState):
return {"messages": [model.invoke([SYSTEM] + state["messages"])]}
builder = StateGraph(MessagesState)
builder.add_node("agent", agent)
builder.add_node("tools", ToolNode(TOOLS))
builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", tools_condition) # tool call -> "tools", otherwise -> END
builder.add_edge("tools", "agent") # tool results go back to the model
graph = builder.compile(checkpointer=MemorySaver())
That is the whole loop from the simulation. tools_condition looks at the model's last message: if it contains tool calls, the graph goes to tools; if not, it ends. ToolNode runs the requested tools and appends their results as tool messages. The checkpointer saves the state after every step, keyed by a thread_id, which gives you conversation memory and the ability to resume.
config = {"configurable": {"thread_id": "customer-42"}, "recursion_limit": 12}
out = graph.invoke({"messages": [("user", "What is 17% of 2,340 euros?")]}, config)
print(out["messages"][-1].content)
For tests you need the answer and the passages the agent actually read. A small helper collects both from the final state:
# app/run.py
from langchain_core.messages import ToolMessage
from app.graph import graph
import uuid
def ask(question: str):
cfg = {"configurable": {"thread_id": str(uuid.uuid4())}, "recursion_limit": 12}
msgs = graph.invoke({"messages": [("user", question)]}, cfg)["messages"]
passages = [p for m in msgs if isinstance(m, ToolMessage) and m.name == "search_docs"
for p in m.content.split("\n\n") if p.strip()]
return msgs[-1].content, passages
The recursion_limit is the agent's budget. Without it, a confused model can call tools forever. Twelve steps is generous for a support agent; set it from what your traces show.
Adding a grading step (corrective retrieval)
The third scenario in the simulation adds a check between search and answer. A small node asks the model whether the retrieved passages answer the question; if not, it rewrites the query and searches once more.
from typing import Literal
from pydantic import BaseModel
class Grade(BaseModel):
relevant: bool
better_query: str
grader = llm.with_structured_output(Grade)
def grade(state: MessagesState) -> Literal["agent", "retry"]:
question = state["messages"][0].content
passages = state["messages"][-1].content
g = grader.invoke(f"Question: {question}\nPassages:\n{passages}\nDo the passages answer the question? "
"If not, propose a better search query.")
return "agent" if g.relevant else "retry"
Wire it as a conditional edge after tools, send retry to a node that calls search_docs with the better query, and cap retries with a counter in the state. The pattern costs one extra model call and removes a whole class of confident wrong answers.
Step 5: tracing with Langfuse
An agent you cannot see is an agent you cannot fix. Langfuse records each run as a trace made of nested observations: spans for steps, generations for model calls, with inputs, outputs, latency, token counts and cost. Set three environment variables and pass the callback handler.
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_HOST="https://cloud.langfuse.com" # or your self-hosted URL
from langfuse import get_client
from langfuse.langchain import CallbackHandler
langfuse = get_client()
handler = CallbackHandler()
out = graph.invoke(
{"messages": [("user", "How long are refurbished laptops covered?")]},
{"configurable": {"thread_id": "customer-42"}, "callbacks": [handler],
"metadata": {"langfuse_session_id": "customer-42", "langfuse_tags": ["support-agent"]}},
)
langfuse.flush() # short scripts exit before the background upload otherwise
Every node, model call and tool call now appears in Langfuse as the waterfall you saw in the simulation. Three views pay back the setup within a week:
- Slow traces. Sort by latency to find the step that dominates: usually a long prompt or a retry loop.
- Cost per session. Tokens times price per model, grouped by user or feature.
- Failed tool calls. Filter on error status to see which tool contracts the model misunderstands.
Langfuse is open source and can be self-hosted, which matters when traces contain customer data. Its Python SDK is built on OpenTelemetry, so traces can also flow to an existing observability stack. Check the SDK major version you install: the v3 and v4 SDKs changed some trace-level APIs, and the Langfuse docs list the current names.
Step 6: unit tests for answers with DeepEval
Tracing shows what happened. Tests decide whether it was good enough to ship. DeepEval works like pytest: a test case holds the input, the agent's output and, for RAG, the retrieved context; metrics score it, usually with an LLM as judge; the test fails below a threshold.
# tests/test_agent.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric, GEval
from app.run import ask # returns (answer, retrieved_passages)
correctness = GEval(
name="Policy correctness",
criteria="The answer states the same rule as the expected output and cites a source id in brackets.",
evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
threshold=0.7,
)
def test_refurbished_warranty():
answer, passages = ask("How long are refurbished laptops covered?")
case = LLMTestCase(
input="How long are refurbished laptops covered?",
actual_output=answer,
expected_output="12 months, batteries 6 months [refurbished §3]",
retrieval_context=passages,
)
assert_test(case, [AnswerRelevancyMetric(threshold=0.7), FaithfulnessMetric(threshold=0.8), correctness])
Run it with deepeval test run tests/test_agent.py locally and in CI. Start with 20 to 50 real questions from support logs, each with the expected answer written by someone who knows the policy. That set is the agent's contract: every prompt change, model swap or retriever tweak must keep it green.
DeepEval also has agent-specific metrics. Tool correctness checks that the expected tools were called; task completion asks a judge whether the agent finished what the user wanted. Use them once the basic answer tests are stable.
Step 7: retrieval scores with RAGAS
DeepEval tells you an answer failed. RAGAS helps you see whether retrieval or generation caused it. Its core metrics split the pipeline:
- Context precision: are the relevant passages ranked at the top of what was retrieved?
- Context recall: did retrieval find everything the reference answer needs?
- Faithfulness: is every claim in the answer supported by the retrieved passages?
- Response relevancy: does the answer address the question that was asked?
# eval/ragas_eval.py
from ragas import evaluate, EvaluationDataset
from ragas.metrics import Faithfulness, LLMContextRecall, LLMContextPrecisionWithReference, ResponseRelevancy
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from app.run import ask
rows = []
for q, ref in [("How long are refurbished laptops covered?", "12 months; batteries 6 months."),
("Can I return a damaged item?", "Yes, full refund or replacement after sending a photo.")]:
answer, passages = ask(q)
rows.append({"user_input": q, "response": answer, "retrieved_contexts": passages, "reference": ref})
result = evaluate(
dataset=EvaluationDataset.from_list(rows),
metrics=[Faithfulness(), LLMContextRecall(), LLMContextPrecisionWithReference(), ResponseRelevancy()],
llm=LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini")),
embeddings=LangchainEmbeddingsWrapper(OpenAIEmbeddings()),
)
print(result.to_pandas())
Read the four numbers together. Low recall with high faithfulness means the model behaved but never saw the right passage: fix chunking, embeddings or add hybrid search. High recall with low faithfulness means the passage was there and the model drifted: fix the prompt or the model. RAGAS releases have moved metric imports between versions, so pin the version in your lock file and check the RAGAS docs when you upgrade.
Step 8: put the loop together
The four tools form one feedback loop:
- Build the graph and run it on real questions.
- Trace every run in Langfuse; find slow, expensive or failing steps.
- Score retrieval with RAGAS on a fixed question set to locate the weak half.
- Test answers with DeepEval in CI so a fix never breaks something else.
- Feed back: questions that fail in production traces become new test cases.
Langfuse can also store datasets and run experiments, and RAGAS or DeepEval metrics can be attached to production traces as scores. Many teams start by running the evaluations offline and add online scoring once the offline set is stable.
Production checklist
- Budget every run. A recursion limit, a timeout per tool, and a maximum spend per session.
- Guard the tools. Validate arguments; never run model output as code or shell; make write actions require confirmation.
- Keep state durable. Swap
MemorySaverfor a database checkpointer (Postgres or SQLite) so runs survive restarts. - Version everything. Prompt, model name, tool schemas and retriever settings go into the trace metadata, so a trace can be reproduced.
- Watch drift. Re-run the evaluation set weekly and on every model change; providers update models behind the same name.
- Protect data. Self-host Langfuse or mask personal data before it is traced.
Common mistakes
Testing only the happy path. Agents fail at the edges: ambiguous questions, missing documents, tool errors. Put those in the test set on purpose.
Using the same model as agent and judge. A judge model tends to approve answers written in its own style. Use a different, stronger model for evaluation where you can, and spot-check the judge against human labels.
Optimising the answer before the retrieval. If context recall is low, no prompt will fix the answer. Measure retrieval first.
No stop condition. A loop without a budget turns one bad tool response into a large bill.
Frequently asked questions
Do I need LangGraph to build an agent?
No. An agent loop is a few dozen lines in plain Python. LangGraph earns its place when you need durable state, checkpoints, human approval steps, branching and parallel steps, because it gives those a tested structure.
What is the difference between DeepEval and RAGAS?
DeepEval is a testing framework: test cases, many metrics and pytest-style assertions for CI. RAGAS focuses on measuring retrieval-augmented generation, with metrics that separate retrieval quality from answer quality. Many teams use both.
Is Langfuse only for LangChain and LangGraph?
No. It has a decorator and SDKs for plain Python and TypeScript, integrations for several frameworks, and accepts OpenTelemetry traces.
How many test questions do I need?
Start with 20 to 50 real questions with expected answers. Grow the set from production failures; a few hundred well-chosen cases cover most support agents.
Can I run all of this without sending data to a cloud?
Yes. Use a local model server, self-host Langfuse, and point the DeepEval and RAGAS judge models at a local or private endpoint. Judge quality drops with very small models, so check a sample by hand.
Summary
An agent is a loop with a state, tools and a stop condition. LangGraph gives the loop a structure, Langfuse makes every step visible, RAGAS shows whether retrieval or generation is the weak half, and DeepEval turns quality into tests that guard every change. Build them in that order, keep a fixed question set, and let production traces feed new tests.
Sources
Continue learning
Cite this article
Kelahmet, Ö. C. (2026, November 17). Building an agentic AI system from scratch with LangGraph, Langfuse, DeepEval and RAGAS. Özer Cem KELAHMET. https://www.ozercemkelahmet.com/articles/build-agentic-ai-from-scratch-langgraph-langfuse-deepeval-ragas
BibTeX
@online{kelahmet2026build,
author = {Kelahmet, {\"O}zer Cem},
title = {{Building an agentic AI system from scratch with LangGraph, Langfuse, DeepEval and RAGAS}},
year = {2026},
month = {nov},
url = {https://www.ozercemkelahmet.com/articles/build-agentic-ai-from-scratch-langgraph-langfuse-deepeval-ragas}
}How this article connects
4 pages link to or from it. Drag to turn, click a dot to open it.
Version historyfirst version
- Published
Factual corrections are listed here with what changed. Wording and formatting edits are not listed.