Agentic RAG

Overview

Agentic RAG is a form of retrieval-augmented generation in which a language model acts as an agent. Instead of running one fixed search and then answering, the agent decides what to retrieve, which source to ask, whether the evidence is good enough, and whether to search again. The loop ends only when the agent judges that it can answer, or when a budget runs out.

This matters because many real questions cannot be answered with a single lookup. They need several steps, several sources, or a second try after a poor first result. Agentic RAG pays for that flexibility with more model calls, more latency and more ways to fail. It is the right tool when questions are hard and varied. It is the wrong tool when a plain pipeline already works. See Naive RAG for the simple baseline and Advanced RAG for the middle ground.

What it is

Agentic RAG combines two ideas that were developed separately. The first is retrieval-augmented generation, where a model reads retrieved passages before it writes an answer. The second is the agent pattern, where a model chooses actions, observes the results and continues until a goal is met.

In an agentic system the retriever is no longer a fixed step. It becomes one tool among several. The agent may also have a SQL tool, a web search tool, a calculator, a code runner or a call to a second knowledge base. The model reads the question, writes a plan, calls a tool, reads the result and decides what to do next.

The core loop

A typical loop has five moves:

• Understand the question and decide whether retrieval is needed at all.
• Choose a source and write a query for it.
• Read the returned evidence and grade it.
• Decide to answer, refine the query, switch source, or split the question.
• Write the final answer with citations to the evidence it used.

What makes it "agentic"

Three properties separate agentic RAG from a fixed pipeline:

• Autonomy over control flow. The model, not the programmer, decides the order and number of steps.
• Tool choice. The model picks among several retrievers and other tools.
• Self-checking. The model inspects its own retrieved evidence and may reject it.

Think of a research assistant in a library. A fixed pipeline is a clerk who always pulls the top five books on a shelf and hands them over. An agentic system is a researcher who reads the question, walks to the right section, skims a book, realises it is the wrong one, tries another shelf, and only then starts writing.

Degrees of agency

Agency is a dial, not a switch. Many production systems sit in the middle.

• Router only: the model picks one of several fixed pipelines, once.
• Router plus grader: the model also checks results and may retry once.
• Full loop: the model plans, calls tools repeatedly and decides when to stop.
• Multi-agent: several specialised agents hand work to each other. See Multi-agent systems.

Where it came from

The roots are in two lines of work. Retrieval-augmented generation was introduced as a named method in 2020 by Lewis and colleagues, who combined a neural retriever with a sequence-to-sequence generator. For a few years the standard recipe stayed simple: embed the query, fetch the nearest chunks, paste them into the prompt.

In parallel, researchers showed that language models could reason and act in interleaved steps. The ReAct approach, published in 2022, had a model write a thought, take an action such as a search, read the observation and repeat. The same idea appears in The ReAct loop. Tool-use training and function calling then made it practical for models to emit structured calls to external tools.

Steps along the way

• Multi-hop question answering datasets showed that one retrieval pass often fails when the answer needs facts from two documents.
• Iterative retrieval methods fed partial answers back into the next query.
• Adaptive retrieval work, including the Self-RAG paper in 2023, trained models to decide when to retrieve and to critique retrieved passages.
• Corrective approaches, such as the CRAG paper in 2024, added an evaluator that scores retrieved documents and triggers a fallback search when they look weak.
• Agent frameworks such as LangGraph and LlamaIndex made the control loop easy to build and inspect.

By the mid 2020s the phrase "agentic RAG" became a common label for any system where retrieval sits inside an agent loop. The label is loose, so always ask what the loop actually does.

How it works

This section walks through a full agentic RAG run, step by step, with everyday analogies.

Step 1: Decide whether to retrieve

Not every question needs a search. "What is two plus two?" does not. "What changed in the refund policy last quarter?" does. A first decision step, often a short prompt to the model, classifies the question. This saves cost and avoids pulling in irrelevant text that can distract the answer.

Analogy: a person asked a question decides first whether they already know the answer or need to look it up.

Step 2: Plan and decompose

For a complex question the agent writes a plan. "Which team had the higher error rate last month, and what changed in their deployment process?" becomes two sub-questions. One needs a metrics table. The other needs a document search. Planning turns one hard question into several easier ones.

Analogy: a cook reading a recipe lists the shopping first, then cooks. They do not run to the shop in the middle of each step.

Step 3: Route to a source

The agent holds a short description of each tool. It picks the one that fits. A question about numbers goes to a SQL tool. A question about policy wording goes to a document index. A question about a recent event goes to web search. Good tool descriptions matter more than clever prompting here.

Step 4: Write the query

The agent writes a query suited to the chosen source. For a vector index this is natural language. For a keyword index it may be a short set of terms. For SQL it is a statement. Query writing is closely related to Query rewriting, and some systems also use HyDE to embed a drafted answer rather than the raw query.

Step 5: Retrieve and read

The tool returns passages, rows or results. Under the hood this may use hybrid search, which blends vector similarity with keyword scoring such as BM25. It may also apply cross-encoder reranking to put the best passages first.

Step 6: Grade the evidence

The agent asks itself: is this relevant, and is it enough? A grader prompt can return a simple label such as relevant, partial or irrelevant. If the evidence is thin, the agent chooses a next action.

Analogy: a student checks a source before quoting it. If the page is about the wrong topic, they close it and look elsewhere.

Step 7: Iterate or stop

Possible next actions are:

• Refine the query with new terms found in the first results.
• Switch to a different source.
• Break the question further and handle a sub-question.
• Stop and answer, noting what is still unknown.

A step budget and a time budget cap the loop. Without a cap, a confused agent can search forever.

Step 8: Generate with citations

The final call receives the question, the plan, the accumulated evidence and instructions to cite. A good system asks the model to state which passage supports each claim, and to say "not found" when the evidence does not cover the question.

Step 9: Verify

Some designs add a last check. A second model call, or a LLM-as-judge step, tests whether each claim is supported by the cited evidence. Unsupported claims are removed or trigger another retrieval round.

A control flow sketch

def agentic_rag(question, tools, max_steps=6):
    state = {"question": question, "evidence": [], "notes": []}
    for step in range(max_steps):
        action = llm_decide(state, tools)      # returns tool name + args, or "answer"
        if action.name == "answer":
            break
        result = tools[action.name](**action.args)
        grade = llm_grade(question, result)    # relevant / partial / irrelevant
        state["notes"].append((action, grade))
        if grade != "irrelevant":
            state["evidence"].append(result)
    return llm_answer(state)

State and memory

The loop needs somewhere to keep track of what it has tried. Short-term state holds the plan, the queries already issued and the evidence so far. Long-term state may hold user preferences or earlier answers. See Agent memory. Keeping a record of failed queries stops the agent from repeating them.

Key concepts and terms

A worked example

Suppose a support team keeps two stores: a document index with policy pages, and a SQL table of refund tickets. A user asks:

"How many refund requests last month were denied, and does the policy say why a request can be denied?"

Trace of an agentic run

1. Decide: the question needs data and policy text, so retrieval is required.
2. Plan: sub-question A is a count from the tickets table. Sub-question B is a policy lookup.
3. Route A to the SQL tool. Route B to the document index.
4. Query A: a SQL statement that counts denied rows for last month.
5. Query B: "reasons a refund request can be denied".
6. Read results: SQL returns a count. The document index returns three passages.
7. Grade: the count is clear. Two of the three passages are relevant. One covers shipping and is dropped.
8. Answer: the agent writes the count, lists the denial reasons from the policy, and cites each source.

The SQL side

SELECT COUNT(*) AS denied_count
FROM refund_requests
WHERE status = 'denied'
  AND created_at >= DATE '2026-09-01'
  AND created_at <  DATE '2026-10-01';

A grader prompt

You are checking retrieved text for a question.
Question: {question}
Passage: {passage}
Reply with one word: relevant, partial or irrelevant.
Reply relevant only if the passage directly helps answer the question.

What a fixed pipeline would do

A naive pipeline would embed the whole question and fetch five chunks. It would probably return policy text and miss the count entirely, because the count lives in a table that is not in the vector index. The answer would be partly right and would not say that data was missing. The agentic version avoided this by routing each sub-question to the right place.

A rough cost view

The figures are illustrative call counts, not measurements. Your own numbers depend on the model, the tools and the question mix.

Strengths and limitations

Strengths

• Handles multi-hop and multi-source questions that a single search cannot.
• Can recover from a bad first query by trying again.
• Can mix unstructured text, SQL and other tools in one answer.
• Can say "I could not find this" when evidence is weak, if designed to do so.
• Makes the reasoning trail visible, which helps debugging and audit.
• Adapts to new sources by adding a tool and a description.

Limitations

• Higher latency, because each step is a model call plus a tool call.
• Higher cost, because many calls consume many tokens.
• Less predictable. Two runs of the same question can take different paths.
• Harder to test. You must evaluate trajectories, not just final answers.
• Prone to loops, over-searching and drift without budgets.
• Tool descriptions and graders become critical prompts that need care.
• Larger attack surface. Retrieved text can carry instructions aimed at the agent. See Prompt injection defense.

When the extra complexity is not worth it

If questions are simple, the corpus is small and one index covers it, a plain pipeline with good chunking and reranking is usually cheaper and just as accurate. Add agency only after you can show failures that agency fixes.

How it compares with alternatives

These approaches combine well. An agent can hold GraphRAG as one tool, a hybrid index as another and a SQL tool as a third. The agent is the coordinator, and the other methods are instruments it can play.

Agentic RAG and fine-tuning

Fine-tuning teaches a model style and skills. Retrieval supplies facts that change. Agentic RAG does not replace fine-tuning, but it often removes the need to bake facts into weights. It also lets the facts stay current without retraining.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types such as numbers, categories and text, missing values, and sometimes no useful column names. Models in this family are usually pre-trained on large numbers of synthetic or real tables and then use in-context learning: they read a set of labelled example rows and predict for new rows without gradient updates. TabPFN is a well-known example of this idea, and FT-Transformer is a related deep model for tabular inputs.

Agentic RAG relates to these models in several concrete ways.

The agent chooses which rows to show the model

In-context learning on tables depends on the context rows. A large table cannot fit into the context window. An agent can act as a smart selector. It can query a database for rows similar to the target, for rows from the same segment, or for a balanced sample of classes. The retrieved rows then become the context for the tabular model. This is retrieval-augmented prediction, with an agent deciding the retrieval query.

Routing between tools and models

A data question may need a SQL aggregate, a tabular model prediction, or a document lookup. An agent can route among them. "Predict which accounts will churn and explain the policy that applies" needs a prediction tool and a document tool. The tabular model is one tool inside the agent loop.

Handling missing values and schema ambiguity

Tables with missing values or without column names make retrieval hard, because text queries have nothing to match. An agent can first call a profiling tool to learn column types and value ranges. It can then propose likely meanings for columns, which is related to semantic labelling work such as Semantic sector heads. The agent can also call an imputation step, such as Imputation networks, before it asks the model to predict.

Anomaly detection with an agent in the loop

For Anomaly detection, an agent can pull a flagged row, retrieve its nearest normal neighbours, fetch related documents such as change logs, and then write a short explanation of why the row looks odd. The detector produces a score, and the agent turns the score into an explanation with evidence.

Continual learning and freshness

Tabular data drifts. Instead of retraining a model every week, an agent can retrieve the latest rows as context at prediction time. This is a light form of continual adaptation. The model weights stay fixed, and the context supplies recent behaviour. It does not replace proper retraining when the data shape changes a lot.

A tabular walk-through with an agent

Consider a table of customer accounts with columns for age band, plan type, monthly usage, support tickets and a churn flag. Some values are missing and the column names are short codes. A user asks which segment is most at risk and why.

1. The agent calls a profiling tool. It learns that three columns are numeric, two are categorical and one has thirty percent missing values.
2. The agent calls a retrieval tool that returns labelled rows similar to a handful of target accounts, with a balanced class mix.
3. The agent passes those rows as context to a tabular foundation model, which predicts churn risk for each target row.
4. The agent calls a document search for the retention playbook and a SQL tool for segment counts.
5. The agent writes a short answer. It states the predicted risk, the evidence rows used, the missing data caveat and the matching playbook section.

The foundation model never saw the playbook, and the language model never computed the predictions. Each part does what it does best.

Division of labour

Practical limits for tabular agents

• Context rows must be consistent in column order and type, or the tabular model may misread them.
• The agent should not invent column meanings. It should label them as guesses and ask for confirmation when stakes are high.
• Sampling rows for context introduces bias. Log the sampling rule so results can be reproduced.
• Predictions on rows far from the retrieved examples deserve lower trust. Report that distance to the reader.

Where it does not apply

Agentic RAG does not itself learn from tables. It is an orchestration pattern. A tabular foundation model still does the table-level prediction. Treat the agent as the planner and the foundation model as a specialist. For deeper material on how such models are built and measured, see /model/architecture, /model/research and /model/benchmarks.

Risks specific to tables

• Row-level data may be sensitive. The agent must respect access controls on every tool.
• Generated SQL can be wrong or expensive. Use read-only credentials, timeouts and row limits.
• Numbers pasted into prompts can be misread. Pass tables in a clean, consistent format.
• Leakage: retrieving rows that contain the target label for the test row will inflate results. Keep a clean split.

Practical guidance and best practices

Start small

Build a plain pipeline first. Measure it. Add routing only when you see questions that go to the wrong place. Add a grader only when you see bad evidence reaching the answer. Add a full loop only when retries clearly help.

Write strong tool descriptions

The agent chooses tools from their descriptions. State what the tool does, what input it expects, what it returns and when not to use it. Short, concrete descriptions beat long vague ones.

tools:
  - name: policy_search
    description: >
      Searches policy documents. Input: a natural language query.
      Returns up to 5 passages with titles. Use for rules and wording.
      Do not use for counts or numbers.
  - name: tickets_sql
    description: >
      Runs a read-only SQL query on the tickets table.
      Use for counts, sums and dates. Do not use for policy text.

Set budgets and stop rules

• Cap steps, tool calls, tokens and wall-clock time.
• Stop when two rounds add no new relevant evidence.
• Return a clear partial answer when the budget ends.

Keep the retrieval layer strong

An agent cannot fix a weak index. Invest in chunking, metadata, hybrid search and reranking. Tools such as pgvector and Elasticsearch can serve as the retrieval back end, and PostgreSQL can hold both the vectors and the structured data.

Evaluate trajectories

Final answer accuracy is not enough. Also record:

• Whether the agent picked the right tool.
• Whether it retrieved the needed evidence.
• How many steps it used.
• Whether every claim is supported by a cited passage.

Frameworks such as RAGAS provide metrics for retrieval and answer quality, and an LLM-as-judge can score groundedness. Build a small test set of hard questions and rerun it after every change.

Log everything

Store each thought, action, observation and grade. When an answer is wrong, the log shows whether routing, retrieval, grading or generation failed.

Guard the inputs and outputs

• Treat retrieved text as untrusted data, never as instructions.
• Limit which tools the agent may call and with which arguments.
• Require confirmation for any action with side effects.
• Filter results by the user's permissions before they reach the model.

Control cost

• Use a small model for routing and grading, and a stronger one for the final answer.
• Cache repeated queries and tool results.
• Run independent sub-questions in parallel.
• Skip retrieval for questions the model can answer from the conversation.

Design patterns that recur in practice

Several patterns show up again and again in agentic RAG systems. Knowing their names helps teams discuss design choices.

• Query routing: a small model picks one of a few indexes or tools for each question.
• Query decomposition: a hard question is split into sub-questions that are answered one at a time or in parallel.
• Corrective retrieval: a grader scores the evidence, and a weak score triggers a rewrite or a fallback source.
• Self-reflection: the model critiques its own draft and requests more evidence for weak claims.
• Plan and execute: a planner writes the full plan first and an executor follows it, which is easier to audit than free-form loops.
• Reflexion style retry: after a failure, the agent writes a short note on what went wrong and uses it in the next attempt.

A small decision table

Latency tactics

Users notice delay more than they notice architecture. Several tactics keep an agent loop responsive.

• Stream partial progress such as "searching policies" so the wait feels shorter.
• Run independent tool calls at the same time rather than one after the other.
• Use a fast model for classification steps.
• Set a hard time limit and return the best partial answer when it is reached.
• Precompute answers for the most common questions and serve them from a cache.

Prompt structure for the decision step

A decision prompt should be short and strict. It lists the tools, shows the state, and asks for one structured output. Free text invites errors. A JSON schema or function call keeps the output machine readable.

{
  "thought": "The question asks for a count, so I need the SQL tool.",
  "action": "tickets_sql",
  "arguments": {"query": "SELECT COUNT(*) FROM refund_requests WHERE status = 'denied'"}
}

Keep the thought field short. Long thoughts cost tokens and rarely improve the choice. Validate every action against an allow list before running it.

A minimal evaluation record

For each test question, store a row like the following. It makes regressions easy to spot.

question_id: q-0042
expected_tools: [tickets_sql, policy_search]
expected_evidence: [doc-17#p3, table:refund_requests]
final_answer_correct: true
steps_used: 4
unsupported_claims: 0

Governance and review

Because the model decides what to call, review matters more than in a fixed pipeline. Put tool permissions, prompts and budgets under version control. Review changes the same way as code. Run the test set before each release. Keep a human review path for answers that touch money, health, safety or legal rules.

Common mistakes

• Giving the agent too many tools. The model gets confused and picks poorly. Start with two or three.
• Vague tool descriptions that overlap, so the router cannot choose.
• No step budget, which leads to endless loops and surprise bills.
• Grading with the same prompt that generates, so the model agrees with itself.
• Testing only the final answer and ignoring bad trajectories that happened to end well.
• Trusting retrieved text as if it were instructions.
• Letting the agent write unrestricted SQL against production data.
• Ignoring the retrieval quality and hoping the agent will compensate.
• Using agents for simple questions where a fixed pipeline is faster and cheaper.
• Forgetting to tell the agent to say "not found" when evidence is missing, which encourages made-up answers.
• Letting context grow without limit, so later steps lose the early plan.
• Not versioning prompts, so a small edit silently changes behaviour.

Frequently asked questions

What is agentic RAG in one sentence?

Agentic RAG is retrieval-augmented generation where a language model acts as an agent that plans, picks tools and sources, checks the evidence and searches again before it answers.

How is agentic RAG different from normal RAG?

Normal RAG runs a fixed sequence of embed, retrieve and generate. Agentic RAG lets the model choose the sequence, so it can retry, switch sources, split a question or skip retrieval when it is not needed.

Is agentic RAG always more accurate?

No. It helps on multi-hop and multi-source questions, but on simple lookups it adds cost and variance with little gain. Accuracy depends on the quality of the retrieval layer, the tool descriptions and the grading step as much as on the agent loop.

Does agentic RAG need a special model?

Not necessarily. Many systems use a general model with function calling and careful prompts. Some research trains models to decide when to retrieve and to critique passages, but a prompted model with good tools is a common starting point.

How many steps should an agent be allowed to take?

Start with a small cap, often between three and eight steps, and raise it only if evaluation shows that more steps help. A cap protects against loops and keeps cost and latency predictable.

How do I test an agentic RAG system?

Build a set of hard questions with known evidence and expected answers. Measure final correctness, tool choice, evidence recall, groundedness and step count. Rerun the set after every prompt, tool or model change.

Can agentic RAG work with SQL and tables?

Yes. A SQL tool is a common part of an agent, and the agent can also pass retrieved rows to a tabular model as context. Use read-only access, limits and clear schema descriptions so that generated queries stay safe and correct.

What are the main security risks?

The main risks are prompt injection through retrieved text, over-broad tool permissions and leakage of data the user should not see. Treat retrieved content as untrusted, restrict tools, filter by permissions and require approval for risky actions.

Is agentic RAG the same as a multi-agent system?

Not exactly. A single agent with several tools can do agentic RAG. A multi-agent system splits the work among several agents with different roles. Multi-agent designs can be useful for large tasks, but they add coordination cost.

When should I choose GraphRAG instead?

Choose GraphRAG when questions need connected or corpus-wide reasoning over entities and relations. Choose agentic RAG when the main difficulty is planning, routing and retrying. The two can be combined, with graph retrieval as one tool for the agent.

Glossary

• Agent: a model-driven loop that picks actions and reads results.
• Agentic RAG: retrieval-augmented generation controlled by an agent loop.
• Chunk: a piece of a document stored and retrieved as a unit.
• Context window: the amount of text a model can read at once.
• Embedding: a numeric vector that represents the meaning of text.
• Function calling: a structured way for a model to request a tool.
• Grader: a step that scores evidence quality.
• Groundedness: how well claims are supported by evidence.
• Hop: one retrieval step in a chain of lookups.
• In-context learning: learning from examples shown in the prompt, with no weight updates.
• Orchestration: the code that manages the loop, tools and state.
• Planner: a step that breaks a question into sub-questions.
• Reranking: reordering retrieved items with a stronger scorer.
• Router: a step that picks a tool or pipeline.
• Trajectory: the recorded path of one agent run.

Summary

Agentic RAG puts retrieval inside an agent loop. The model plans, routes to the right tool, writes queries, grades the evidence and decides whether to search again or answer. This makes it strong for multi-hop and multi-source questions and weak on cost, latency and predictability.

Use it when a simple pipeline fails in ways that retries and routing can fix. Keep the retrieval layer strong, write clear tool descriptions, set budgets, log every step and evaluate trajectories as well as answers. For tabular work, treat the agent as a planner that selects context rows, calls profiling, imputation and prediction tools, and explains results, while a tabular foundation model does the table-level learning. Explore the related pages on naive RAG, GraphRAG, hybrid search and /model/architecture to go deeper.

Related

It belongs to RAG in my stack, under LLMs, retrieval, and evaluation. Others in the same group: