Modular RAG

Overview

Modular RAG is a way of designing retrieval-augmented generation systems as a set of swappable components. Retrieval, routing, query processing, memory, reranking, fusion, generation and validation are treated as modules with clear inputs and outputs. A given application wires the modules into a flow that fits its data and tasks, and the flow can differ from one question type to the next.

The point of the modular view is flexibility with discipline. Naive RAG is one fixed chain. Advanced RAG is a longer fixed chain. Modular RAG says that the chain itself should be a design choice: linear for some questions, branching for others, looping when needed. This makes it easier to adapt to many data types, to test one part at a time and to swap a component without rewriting the system. Agentic RAG can be seen as the case where a model decides how to wire the modules at run time.

What it is

Modular RAG is an architecture style. It does not prescribe a single pipeline. It prescribes a way of thinking: break the system into parts that each do one job, define how they connect, and let the connections vary.

Modules and operators

Descriptions of modular RAG usually separate two levels.

• Modules: the larger building blocks such as indexing, pre-retrieval, retrieval, post-retrieval, generation and orchestration.
• Operators: the specific techniques inside a module, such as a keyword retriever, a cross-encoder reranker or a query rewriter.

A team picks operators, groups them into modules and connects the modules in a flow. Replacing an operator, say a dense retriever with a hybrid one, should not require changes elsewhere.

Analogy: a kitchen with stations. There is a prep station, a grill, a sauce station and a plating station. A simple order uses two stations. A complex order uses all of them in a chosen sequence. The head cook decides the route for each order, and any station can be upgraded without rebuilding the kitchen.

Common flow patterns

What stays constant

Whatever the flow, each module has an interface. The retriever takes a query and filters and returns scored items. The reranker takes a query and items and returns items in a new order. The generator takes a question and evidence and returns text with citations. Stable interfaces are what make swapping possible.

Where it came from

Modular thinking grew out of the observation that production RAG systems kept sprouting extra parts. After the original 2020 formulation of retrieval-augmented generation, teams added keyword search, rerankers, query rewriters, routers, caches and verifiers. Each addition solved a problem but made the pipeline harder to reason about as one fixed line.

Stages in the literature

Survey writing in 2023 and 2024 described a progression:

• Naive RAG: index, retrieve, generate.
• Advanced RAG: add pre-retrieval and post-retrieval steps.
• Modular RAG: treat the whole system as a collection of modules and flows, including routing, memory, search over multiple sources, fusion and prediction modules.

The modular stage also absorbed ideas from earlier research: iterative retrieval, where generation feeds the next query; adaptive retrieval, where the model decides when to retrieve, as in the Self-RAG work from 2023; and corrective retrieval, as in the CRAG work from 2024.

Software side

Orchestration libraries made modular design practical. LlamaIndex offers composable retrievers, routers and query engines. LangGraph models flows as graphs with state, conditions and loops. These tools turn the idea of modules and flows into code that can be tested and versioned.

How it works

This section describes the parts of a modular RAG system and how a request moves through them, using everyday analogies.

The module catalogue

A modular system draws from a catalogue like the following. A given application uses only the modules it needs.

Step 1: Receive and understand the request

The request enters with its context: user, session, permissions and chat history. A memory module supplies recent turns. A query-processing module rewrites the question into a standalone form.

Analogy: a receptionist who knows who you are, what you asked earlier and which department you need.

Step 2: Route

A router decides what happens next. It can be a simple rule ("if the question contains an order number, use the orders database"), a small classifier, or a language model with a short list of options. Routes may lead to different retrievers, different prompts or even to no retrieval at all.

def route(question, meta):
    if meta.get("has_order_id"):
        return "orders_sql"
    label = classify(question)  # "policy", "product", "chitchat", ...
    return {"policy": "policy_index",
            "product": "product_index",
            "chitchat": "no_retrieval"}.get(label, "default_index")

Step 3: Retrieve from one or more sources

The chosen path calls its retriever or retrievers. Sources may be vector indexes, keyword engines, SQL databases, graph stores such as the ones behind GraphRAG, or web search. Calls to independent sources can run in parallel.

Step 4: Fuse and refine

If several sources respond, a fusion module merges the lists. A reranker then orders the merged candidates by relevance to the question, and a filter removes anything the user is not permitted to read.

Step 5: Decide whether the evidence is enough

A validation module can grade the evidence. If it is weak, the flow can loop: rewrite the query, switch source or widen the search. A retry limit prevents endless loops.

Step 6: Generate

The generator receives the question, the memory and the evidence, and writes an answer. The prompt template and the output format belong to this module and can be changed without touching retrieval.

Step 7: Validate and respond

A last module checks that claims are supported by cited evidence, that the answer follows policy and that the output matches the required format. If a check fails, the flow can regenerate or return a safe fallback message.

Step 8: Update memory and logs

The turn is stored so the next question has context. Logs capture each module's input and output, which supports debugging and evaluation.

Flow as configuration

A strength of the modular style is that the flow can live in configuration rather than in code. That makes it easy to compare designs and roll back.

flow: support-assistant-v3
modules:
  memory: {type: sliding_window, turns: 6}
  rewrite: {type: llm, keep_entities: true}
  router:
    type: classifier
    routes:
      policy: policy_index
      product: product_index
      billing: billing_sql
      smalltalk: none
  retrievers:
    policy_index: {type: hybrid, top_k: 40}
    product_index: {type: dense, top_k: 30}
    billing_sql: {type: sql, read_only: true, row_limit: 100}
  rerank: {type: cross_encoder, top_n: 6}
  validate: {type: faithfulness, on_fail: retry_once}
  generate: {type: llm, cite: true, refuse_if_unsupported: true}
limits: {max_steps: 6, timeout_seconds: 20}

Interfaces that make swapping possible

Each module should accept and return plain, documented data. For example, every retriever might return a list of items with an id, text, source, score and metadata. If all retrievers share that shape, a reranker works with any of them.

from dataclasses import dataclass, field

@dataclass
class Item:
    id: str
    text: str
    source: str
    score: float
    meta: dict = field(default_factory=dict)

class Retriever:
    def search(self, query: str, filters: dict, top_k: int) -> list[Item]:
        raise NotImplementedError

class Reranker:
    def rank(self, query: str, items: list[Item], top_n: int) -> list[Item]:
        raise NotImplementedError

Key concepts and terms

A worked example

A company runs an internal assistant for employees. It must answer questions about HR policy, product documentation, and the status of support tickets. The data lives in three places: a document store, a product wiki and a ticket database.

Start with a fixed pipeline

A first version uses one index that mixes everything, with a naive pipeline. Results are mixed. Ticket questions fail, because ticket data is in a database. Policy questions return product pages. Follow-ups lose context.

Modularise

The team splits the system into modules:

1. Memory: keep the last six turns.
2. Rewrite: make the question standalone.
3. Router: classify into policy, product, tickets or smalltalk.
4. Retrievers: a hybrid retriever for policy, a dense retriever for the wiki, a read-only SQL tool for tickets.
5. Rerank: a cross-encoder for text results.
6. Validate: a faithfulness check with one retry.
7. Generate: a template that cites sources and refuses when unsupported.

Three requests, three paths

A follow-up with a loop

A user asks, "Which plan includes audit logs, and how do I enable them?" The router sends it to the product index. The validator sees that the retrieved passages cover enabling but not the plan. It triggers one retry with a rewritten query focused on plans. The second round finds the plan table, and the answer covers both parts.

Ablation: which module earns its place?

The team runs the test set with and without each module.

The table shows the kind of conclusion an ablation gives. Real numbers come from your own data. A module that shows no gain is removed to save cost.

Swapping a component

Six months later, a better embedding model is available. Because the retrievers sit behind a shared interface, the team re-indexes the product wiki with the new model and compares it on the test set. The change touches one retriever and one config line. Nothing else moves.

Adding a source

A new knowledge base appears. The team adds a retriever, one routing label and a test group. The rest of the system is unchanged. In a monolithic pipeline, this change would have meant reworking the prompts and the index together.

Strengths and limitations

Strengths

• Adapts to many data types and question types in one system.
• Each module can be tested, replaced and improved on its own.
• Different questions can follow different paths, which saves cost on easy ones.
• Supports routing, memory, fusion and validation as first-class parts.
• Configuration-driven flows make experiments and rollbacks simple.
• Gives clear places to attach logging, access control and budgets.
• Scales with team size: different people can own different modules.

Limitations

• More design work up front. Interfaces, configs and tests take time.
• More parts mean more failure modes and more things to monitor.
• Routers can misroute, and a wrong route can fail silently.
• Flexibility can turn into over-engineering. Teams may build modules they never need.
• Debugging flows with branches and loops is harder than debugging one line.
• Latency can grow with each added module.
• Evaluating the whole system needs both module-level and end-to-end tests.

Guarding against over-engineering

A useful rule: do not add a module until you can point to a failure that it fixes and a metric that will show the fix. Keep a list of modules that are switched off, and remove those that stay off for months.

How it compares with alternatives

Modular RAG versus advanced RAG

Advanced RAG is a list of techniques applied in a fixed order. Modular RAG is the architecture that lets you decide the order, skip steps, branch and loop. An advanced pipeline can be expressed as one flow in a modular system.

Modular RAG versus agentic RAG

In modular RAG the designer defines the possible flows and often the rules for choosing among them. In agentic RAG a model chooses the next action each step. The two meet in the middle: a model-based router inside a modular system is a small dose of agency. See also multi-agent systems for designs where several agents take roles.

Modular RAG and frameworks

Frameworks help but do not replace design. LlamaIndex provides routers, query engines and retrievers that compose. LangGraph provides graph-shaped control flow with state and loops. Pick one that matches your team's habits and keep interfaces thin, so that you can switch.

Storage layer choices

A modular system often uses more than one store: a vector index, a keyword engine, a relational database and sometimes a graph store. Options include pgvector, Elasticsearch, Qdrant, Weaviate, Milvus, Pinecone and PostgreSQL. Keeping them behind a retriever interface lets the rest of the system ignore which one answers.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types such as numbers, categories and text, missing values and sometimes no useful column names. These models are usually pre-trained on large numbers of synthetic or real tables and then use in-context learning: they read labelled example rows in the prompt and predict for new rows without updating weights. TabPFN is a well-known example, and FT-Transformer is a related deep model for tabular inputs.

Modular RAG is about structure and wiring, so it maps naturally onto data systems that include such models.

The tabular model as a module

In a modular system, a tabular foundation model is one more module with a clear interface: it takes context rows and target rows, and returns predictions with uncertainty. Other modules can sit around it:

Because the modules share interfaces, a team can swap the imputation method, change the row selector or try a different prediction model without redoing the rest.

Routing between tools and models

Different questions need different modules. "How many rows have status closed?" is a SQL job. "Which customers are likely to leave?" is a prediction job. "What does the discount field mean?" is a document job. "Which rows look wrong?" is an anomaly job. A router picks the path. Semantic labelling work, such as semantic sector heads, can help a router recognise what kind of table it is looking at, even when column names are absent or cryptic.

Selecting context rows

In-context learning depends on which labelled rows are placed in the prompt. A row-retrieval module can combine several signals: exact matches on key categories, distance on numeric features, recency and class balance. This is the tabular cousin of hybrid retrieval and reranking. The module's output is a compact, clean context for the prediction module.

Handling missing values and absent names

A profiling module can report that a column is mostly empty or that a numeric column behaves like a category. A schema module can propose likely meanings and mark them as guesses. An imputation module can fill gaps before prediction, or the tabular model can be run with the gaps left in place if it handles them natively. The modular layout makes the choice a configuration flag that can be tested.

Continual adaptation

Tabular data drifts. Modular flows can refresh the row retrieval index on a schedule so that new behaviour reaches the context without retraining the model weights. A validation module can compare recent performance with a reference and raise an alert when the shape of the data changes enough to justify retraining.

Anomaly detection with explanation

An anomaly module scores rows. A document-retrieval module fetches related notes. A generation module writes a short explanation with citations. A validation module checks that the explanation does not claim more than the evidence shows. Each module can be improved separately.

Honest limits

• Modular RAG does not learn from tables. It organises the pieces around a model that does.
• More modules mean more places for leakage. Keep a strict rule that context rows never include the label of the target row.
• Access control must apply in every retrieval module, especially for row-level data.
• Text-based retrievers are a weak fit for numeric similarity. Use retrieval methods that match the data type.

For how tabular foundation models are built and measured, see /model/architecture, /model/research and /model/benchmarks.

Practical guidance and best practices

Begin with the simplest flow that works

Build a naive pipeline, measure it, then modularise only where a failure demands it. A modular system with one active flow is fine. The structure is there when you need it.

Define interfaces first

Write down the input and output of each module before coding it. Use simple, serialisable data. Version the interfaces so that old logs stay readable.

Make flows data, not code

Store the flow in configuration with a version. Review changes like code. Keep the last good version for fast rollback.

Test at three levels

Use RAGAS or TruLens style metrics for retrieval and answer quality, and an LLM-as-judge for scale, with human spot checks.

Evaluate the router on its own

Create a labelled set of questions with the correct route. Measure routing accuracy and look at the confusion between routes. A small router error rate can be the largest source of end-to-end failures.

Plan for failure

• Give each module a timeout.
• Define a fallback for each module, for example a simpler retriever if the hybrid one fails.
• Return a clear message when no path succeeds, rather than a guess.
• Cap the total steps, tokens and time per request.

Secure every module

• Apply access filters in each retriever, not only at the end.
• Treat retrieved text as data and not as instructions. See prompt injection defense.
• Add guardrails at input and output boundaries.
• Restrict SQL and other tool modules to read-only access with limits.

Manage memory carefully

Memory helps follow-ups but can also carry errors forward. Keep a short window, summarise old turns, and give users a way to reset. Do not store sensitive data longer than needed. See agent memory for related ideas.

Observe the system

Log each module's input, output, latency and errors with a request id. Build a dashboard of route frequencies, retries, empty results and refusal rates. Sudden shifts often show a data or model change.

Keep cost in view

• Send easy questions down cheap paths.
• Use small models for routing, rewriting and validation.
• Cache stable results.
• Run independent retrievals in parallel.

A migration path from a monolith

Moving an existing pipeline to a modular layout works best in small, reversible steps.

1. Wrap the current retriever, reranker and generator behind the interfaces above, with no change in behaviour.
2. Move prompts and settings into versioned configuration.
3. Add logging at every module boundary.
4. Add a router with a single default route, then add routes one at a time, each with its own test group.
5. Remove modules that show no gain in ablation tests.

Questions to ask before adding a module

• What failure does it fix, and how will I see the fix in a metric?
• What does it cost in latency, tokens and upkeep?
• What happens when it fails or times out?
• Who owns it, and how will it be tested?

Common mistakes

• Building many modules before measuring a single baseline.
• Skipping interface definitions, so modules become tangled.
• Hard-coding flows in scattered code instead of one versioned configuration.
• Trusting the router without measuring it.
• Giving the router too many options with overlapping descriptions.
• Forgetting fallbacks, so one failing module stops the whole request.
• Letting loops run without a step or time limit.
• Testing only end to end, which hides which module broke.
• Applying access control only at the end of the flow.
• Adding memory without a plan to limit or clear it.
• Switching embedding models in one retriever without re-indexing and re-testing.
• Treating modular as a goal in itself rather than a means to adapt and test.
• Keeping modules that never improve a metric.
• Ignoring data-type fit, for example using text similarity to compare numeric rows.

Frequently asked questions

What is modular RAG?

Modular RAG is an architecture style that treats retrieval-augmented generation as a set of swappable modules, such as query processing, routing, retrieval, reranking, memory, generation and validation. A flow wires them together, and the flow can differ by question type.

How is modular RAG different from advanced RAG?

Advanced RAG adds pre-retrieval and post-retrieval techniques to a fixed chain. Modular RAG goes further by making the chain itself configurable, with routes, branches and loops. An advanced pipeline can be one flow inside a modular system.

Is modular RAG the same as agentic RAG?

No. In modular RAG the designer defines the modules and the allowed flows. In agentic RAG a model chooses actions step by step. A model-based router inside a modular design blends the two, but the designer still sets the boundaries.

Do I need a framework to build modular RAG?

No, but frameworks help. Libraries offer routers, retrievers and graph-style control flow so that you do not write everything from scratch. The key is to keep module interfaces thin and well documented, so that you can change tools later.

What is the role of a router?

A router decides which path, source or tool handles a request. It may use rules, a classifier or a language model. A good router saves cost by sending easy questions down simple paths and improves accuracy by choosing the right source.

How do I know which modules to keep?

Run ablation tests. Remove one module at a time and compare results on a real test set. If removing a module does not hurt quality, latency or safety, it is a candidate for removal.

Does modular design make systems slower?

It can, if many modules run in series. Good design sends simple requests down short paths, runs independent steps in parallel and uses small models for control steps. Measure latency per module to find the real cost.

How does memory fit into modular RAG?

A memory module stores earlier turns or notes and feeds them to query rewriting and generation. It lets follow-up questions work. Keep it short, summarise old content, avoid storing sensitive data without need and give users a way to reset it.

Can modular RAG handle tabular data?

Yes, as an organising layer. A tabular foundation model, an imputation step, an anomaly scorer, a row retriever and a document retriever can each be a module. A router sends each question to the right path. The table-level learning is done by the tabular model, not by the RAG layer.

How do I test a modular RAG system?

Test at three levels: each module against its contract, each route on its own question types, and the whole system end to end. Measure router accuracy separately, because a wrong route can look like a retrieval or generation failure.

Glossary

• Ablation: a test that removes one part to measure its effect.
• Adaptive retrieval: deciding per question whether and how much to retrieve.
• Fallback: a backup action used when a step fails.
• Flow: the chosen connection of modules for a request.
• Fusion: merging ranked lists from several retrievers.
• Interface contract: the agreed input and output shape of a module.
• Iterative retrieval: using partial results to shape the next search.
• Memory module: storage of earlier turns or notes.
• Module: a building block with one job.
• Operator: a specific technique inside a module.
• Orchestration: the logic that runs modules in order with conditions and loops.
• Reranker: a model that reorders candidates by relevance.
• Router: a component that picks a path or source.
• Validation: a check on evidence or answers before they are returned.
• Versioned configuration: a stored, reviewable description of a flow.

Summary

Modular RAG treats a retrieval-augmented system as a set of modules with clear interfaces, connected by flows that can be linear, conditional, branching or looping. Routing, memory, fusion, reranking, generation and validation become parts you can test, swap and tune on their own. This suits applications with several data types and question types.

Start with the simplest flow that works, define interfaces early, store flows as versioned configuration and test at module, flow and end-to-end levels. Measure the router, add fallbacks and limits, and apply access control in every retriever. For tabular work, the tabular foundation model, imputation, anomaly scoring, row retrieval and document retrieval each become modules that a router can combine. Continue with Agentic RAG, Advanced RAG, LangGraph and /model/architecture.

Related

It belongs to RAG in my stack, under LLMs, retrieval, and evaluation. Others in the same group: