Multi-agent RL

Overview

Multi-agent reinforcement learning (MARL) is reinforcement learning with more than one learner acting in the same environment. Each agent observes part of the world, picks actions, and receives rewards. The twist is that the other agents are learning too, so the world each agent faces keeps changing while it learns. That single fact explains most of what makes MARL harder than the single-agent case.

MARL is used when a problem naturally has several decision makers: machines on a production line, vehicles sharing roads, bidders in an auction, or robots in a warehouse. Agents can cooperate toward a shared goal, compete in a zero-sum game, or sit somewhere in between. This article explains the idea from the ground up, shows a small worked example, compares MARL with its neighbors, and ends with a section on how it relates to models that learn from raw tables.

What it is

In ordinary reinforcement learning, one agent interacts with an environment. It sees a state, takes an action, gets a reward, and the environment moves to a new state. The agent tries to maximize the total reward it collects over time.

In multi-agent RL, the environment contains N agents. Each agent has its own observation, its own action set, and often its own reward. The next state depends on the joint action of all agents, not on one agent alone.

The formal model

The standard formal object is the Markov game, also called a stochastic game. It extends the Markov decision process with several players. It is defined by:

• A set of states.
• A set of actions for each agent.
• A transition rule that maps a state and a joint action to a distribution over next states.
• A reward function for each agent.
• A discount factor that makes near-term reward count more than distant reward.

When agents cannot see the full state, the model becomes a partially observable Markov game. When all agents share one reward and have partial views, it is often called a decentralized partially observable Markov decision process, or Dec-POMDP.

Three relationship types

Most real problems are mixed. Two machines on a line both want the line to run, but each also wants to avoid the blame for a jam.

Where it came from

The roots of MARL lie in two older fields. One is game theory, which studies strategic choices among rational players and gave us the idea of an equilibrium. The other is single-agent reinforcement learning, which grew from dynamic programming and trial-and-error learning in psychology.

Early work in the 1990s brought the two together. Researchers extended Q-learning to Markov games and studied what happens when several learners adapt at once. Minimax Q-learning, for zero-sum games, is a well-known early example. Later work studied general-sum games, team games, and the question of whether independent learners converge at all.

Deep learning changed the scale of the field in the 2010s. Deep Q-networks showed that neural networks could learn control from raw input in one-agent settings. Researchers then applied the same ideas to several agents. Papers on independent deep Q-learning, on centralized critics for continuous control, and on value decomposition for teams set the main patterns still used today. Self-play systems that mastered board games and large video games showed what could happen when agents train against copies of themselves at scale.

A few lines of research stand out:

• Independent learners: each agent runs its own single-agent algorithm and treats others as part of the environment.
• Centralized training with decentralized execution: use extra information during training, but let each agent act on its own at run time.
• Value decomposition: split a team value into per-agent parts so each agent can learn from a team reward.
• Population and league training: train against a diverse pool of opponents to avoid fragile strategies.
• Communication learning: let agents learn what messages to send to each other.

How it works

Think of a group of drivers entering a busy roundabout. Each driver learns by experience. If everyone changes habits at the same time, no driver can rely on what worked yesterday. This is the heart of MARL.

Step 1: Agents observe

Each agent receives an observation. It may be the full state or only a slice of it, like a driver who sees only the cars ahead. Partial views are normal, so many methods give agents a memory, such as a recurrent network, to summarize recent history.

Step 2: Agents act together

Each agent picks an action from its own policy. The environment applies the joint action. Timing can be simultaneous or take turns. Board games are turn based. Traffic is simultaneous.

Step 3: The world responds

The environment returns a new state and rewards. In a cooperative game every agent gets the same team reward. In a competitive game the rewards are opposed. In a mixed game they differ.

Step 4: Agents learn

Each agent updates its policy or value function from the experience. Here the main design choices appear.

Independent learning

Every agent learns as if alone. This is simple and scales well. The problem is non-stationarity. From agent A's point of view, agent B is part of the environment, and B's behavior keeps changing. The rules that A is trying to learn will not hold still, which breaks the guarantees that single-agent methods rely on.

Centralized training, decentralized execution

During training, a learner may see everything: all observations, all actions, even hidden state. A centralized critic judges the joint situation. At run time each agent uses only its own observation and a small policy. Think of a sports team that reviews game film together, then plays on the field without a coach shouting every move.

Value decomposition

In cooperative games the team reward is hard to attribute. If a team wins, which player helped? Value decomposition methods represent the team value as a combination of per-agent values, often with a constraint that keeps the combination monotone. Each agent can then pick its best action locally, and the team still improves.

Self-play

In competitive games, an agent can train against a copy of itself. As it improves, its opponent improves. This produces a curriculum without human data. The risk is cycling, where strategy A beats B, B beats C, and C beats A, so the agent never settles. League methods keep older opponents around to prevent this.

Step 5: Repeat until a stable pattern appears

The goal is often an equilibrium. In a Nash equilibrium, no agent can gain by changing its own strategy while others stay put. Learning methods may or may not reach one. In practice teams look for a policy that performs well, and is hard to exploit, across many partners.

The credit assignment problem

When ten agents share one reward, each agent receives a noisy signal. It cannot tell whether the reward changed because of its own action or someone else's. This is called multi-agent credit assignment. Remedies include:

• Counterfactual baselines that ask, "what would the team have scored if I had acted differently?"
• Difference rewards that subtract the value of a default action.
• Value decomposition, as described above.
• Reward shaping that gives each agent a local signal.

Common algorithm families

Many named methods fit into a few families. Knowing the family tells you what problem it was built for.

Value-based methods

• Each agent keeps a table or network that scores actions.
• Independent Q-learning is the simplest member.
• Value decomposition methods are the team version.
• They suit discrete actions and cooperative rewards.

Policy-gradient and actor-critic methods

• The agent learns the policy directly.
• A critic estimates how good the situation is.
• In multi-agent form, the critic may see all agents during training.
• They suit continuous actions, such as steering or throttle.

Equilibrium-seeking methods

• These aim to find a Nash or correlated equilibrium.
• They are common in competitive and general-sum games.
• Fictitious play and regret minimization belong here.
• They give strong guarantees in small games and approximate ones in large games.

Model-based and planning methods

• The agent learns a model of the world and of the other agents.
• It then plans ahead using that model.
• Opponent modeling is a popular add-on: the agent guesses what others will do next.

Communication-based methods

• Agents learn a message protocol along with the policy.
• Messages can be continuous vectors or discrete symbols.
• The channel may be limited by bandwidth, which forces useful, short messages.

Everyday analogies for the main ideas

• Non-stationarity is like learning to play tennis while your partner keeps changing their serve.
• Centralized training is a coach who watches the full match and gives notes afterward.
• Decentralized execution is the players making their own calls on the court.
• Value decomposition is splitting a restaurant bill by what each person ordered.
• Self-play is practicing against a mirror that gets better as you do.
• Credit assignment is a group project where only one grade is given.
• Emergent behavior is a flock of birds forming a shape that no single bird planned.

Key concepts and terms

A worked example

Consider two machines, A and B, on a short production line. Each step, each machine chooses to RUN or WAIT. There is one shared buffer between them with space for two items. If A runs and the buffer is full, the line jams and the team loses reward. If B runs and the buffer is empty, B wastes energy. If A fills the buffer and B drains it at a good rate, the team earns reward for each finished item.

This is a cooperative game with a shared reward. The sketch below uses independent Q-learning in a tiny table. It is meant to show the shape of the loop, not a production system.

import random
from collections import defaultdict

ACTIONS = ["RUN", "WAIT"]

def step(buffer, a_act, b_act):
    reward = 0.0
    if a_act == "RUN":
        if buffer >= 2:
            reward -= 2.0          # jam
        else:
            buffer += 1
    if b_act == "RUN":
        if buffer == 0:
            reward -= 0.5          # wasted run
        else:
            buffer -= 1
            reward += 1.0          # finished item
    return buffer, reward

q_a = defaultdict(float)  # key: (buffer, action)
q_b = defaultdict(float)

def choose(q, s, eps):
    if random.random() < eps:
        return random.choice(ACTIONS)
    return max(ACTIONS, key=lambda a: q[(s, a)])

alpha, gamma, eps = 0.1, 0.9, 0.1
buffer = 0
for t in range(50000):
    s = buffer
    a = choose(q_a, s, eps)
    b = choose(q_b, s, eps)
    buffer, r = step(buffer, a, b)
    s2 = buffer
    best_a = max(q_a[(s2, x)] for x in ACTIONS)
    best_b = max(q_b[(s2, x)] for x in ACTIONS)
    q_a[(s, a)] += alpha * (r + gamma * best_a - q_a[(s, a)])
    q_b[(s, b)] += alpha * (r + gamma * best_b - q_b[(s, b)])

What to expect

Early on, the machines jam the line often, because each explores at random. Over time machine A learns to wait when the buffer is full. Machine B learns not to run when the buffer is empty. The pair settles on a rhythm that keeps the buffer around one item.

A numeric walk-through

Suppose the buffer holds 1 item. A chooses RUN and B chooses RUN. A adds one item, the buffer is 2. B removes one item, the buffer is 1, and the team reward is +1.0.

For A, the Q-update for state 1 and action RUN is:

• Old value: 0.0
• Reward: 1.0
• Best next value: 0.0
• New value: 0.0 + 0.1 * (1.0 + 0.9 * 0.0 - 0.0) = 0.1

B performs the same update. Both agents nudge their tables upward. Notice that A cannot tell whether B's action helped. If B had waited, A's RUN would have produced 0 reward. This is credit assignment in miniature, and it is why joint training with a central critic is often preferred for larger teams.

Strengths and limitations

Strengths

• Natural fit for problems with several decision makers.
• Decentralized execution scales to many agents and tolerates single-agent failures better than one giant controller.
• Self-play can generate useful training data without human labels.
• Emergent coordination can find strategies designers did not foresee.
• Parameter sharing lets one small network serve many similar agents.

Limitations

• Non-stationarity breaks many convergence guarantees.
• The joint action space grows exponentially with the number of agents.
• Credit assignment is hard with a shared reward.
• Training is sample hungry, and simulators are often required.
• Policies can overfit to the specific partners seen in training and fail with new ones.
• Results are sensitive to random seeds, reward design and hyperparameters.
• Evaluation is tricky: a good score against one opponent may mean little against another.
• Safety is a concern when learned agents interact with people or physical systems.

How it compares with alternatives

A practical rule: if one decision maker controls everything, use single-agent RL or plain optimization. If several actors with their own information must adapt to each other, MARL becomes useful.

Role in tabular foundation models

Tabular foundation models learn from raw tables: rows as examples, columns as features, mixed numeric and categorical types, missing cells, and often no helpful column names. Examples include TabPFN, SAINT and FT-Transformer, described further in /model/research. MARL does not belong in the core of these models, but it connects to them in several concrete ways.

As a producer of tables

Every MARL run is a data factory. A simulator with many agents logs one row per agent per step: an observation vector, an action, a reward, a done flag. Over thousands of runs this becomes a large table with mixed types, such as continuous sensor readings, categorical machine states and integer counts, plus missing cells when a sensor drops out. A tabular foundation model can learn from such logs directly, with no hand-built features.

As a consumer of tabular predictions

An agent often needs a quick estimate of something. For a line controller, that could be the chance that a machine will fail in the next hour, or the expected cycle time. A tabular model trained on the plant history can supply this as an input to the agent state. Because tabular foundation models do in-context learning, the model can be pointed at a fresh table of recent rows without a long retraining step. This is helpful when a plant changes often.

Synthetic pre-training and simulators

Many tabular foundation models are pre-trained on large numbers of synthetic tables generated from random programs or causal graphs. A multi-agent simulator is a richer kind of table generator. It yields tables with feedback loops, delayed effects and interaction between rows that are not independent. Mixing such simulator tables into pre-training is one way to widen the range of structure a model has seen. This is an idea for research and design, not a claim about a particular released model.

Anomaly detection and robustness

Learned multi-agent policies can fail in odd ways when conditions shift. A tabular model that learns the normal shape of the logged data can flag unusual rows, which helps a human supervisor see when agents leave the operating range. See anomaly detection for the general technique.

Imputation of missing observations

Agents with partial views and sensor gaps create missing cells. Imputation models fill gaps from the other columns of the row. A tabular model with masked-cell training, as in masked cell modeling and imputation networks, can do this, giving agents a fuller picture than their raw sensors.

Continual learning

Plants and markets drift. A policy trained last month may face new conditions. Continual learning methods let a model absorb new tables without forgetting older ones. This pairs with MARL because the agents themselves must keep adapting, and the supporting predictors must keep pace.

Sector and semantic classification

When agents operate across many sites, logs from different sectors use different column conventions. A model that can classify a table's sector or the meaning of a column, even with missing or cryptic names, helps route data to the right agent family. This is tabular semantic understanding rather than control, but it feeds the same pipelines.

How the pieces fit in a data pipeline

A typical pipeline looks like this:

• A simulator or plant produces raw logs, one row per agent per step.
• A cleaning step fixes types, units and timestamps.
• A tabular model fills gaps and flags odd rows.
• A feature view is handed to the agents as extra state.
• Agent outcomes are logged back into the same table format.
• A scheduled job refreshes the tabular model from the newest rows.

In this loop the table is the shared language. Agents speak in actions and rewards, and the tabular model speaks in columns and rows. A stable schema lets both sides evolve without breaking each other.

What to keep honest

It is easy to overstate the link between control and tabular modeling. A tabular foundation model predicts or completes table cells. It does not choose actions on its own. A MARL policy chooses actions but does not summarize history unless it is built to. Using each for what it does best, and testing the joint system end to end, is the sound approach.

See also /model/architecture and /model/benchmarks for how such models are built and measured.

Practical guidance and best practices

Start simple

• Begin with one agent on a simplified version of the task. Make sure the reward is sensible before adding agents.
• Add agents one at a time and watch how performance changes.
• Try independent learners first. Move to centralized critics only if coordination fails.

Design the reward with care

• Prefer a reward that reflects the real goal, then add small shaping terms if learning stalls.
• Watch for reward hacking, where agents find loopholes that raise the score but break the real objective.
• For cooperative tasks, test both team reward and individual reward, and compare.

Use parameter sharing for similar agents

If agents are interchangeable, share one network and add an agent identifier to the input. This cuts sample needs sharply and keeps behavior consistent.

Train against diverse partners

• Mix in older versions of your own policies.
• Randomize partner behavior during training.
• Evaluate with partners the agent has never met.

Measure properly

• Run multiple random seeds and report spread, not just the best run.
• Track exploitability in competitive games, meaning how much a best-response opponent can win.
• Log team and individual returns separately.
• Keep a fixed evaluation set of scenarios.

Treat simulators as data sources

Store every rollout in a clean tabular format with a stable schema. Include agent id, step, observation columns, action, reward and episode id. Good tables make later analysis, anomaly checks and tabular modeling easy.

Plan for safety

• Add hard constraints outside the learned policy, such as limits on speed or force.
• Use a fallback controller that takes over when the learned policy leaves its safe region.
• Test in simulation before any real deployment.

Where MARL is applied

Evaluation checklist

• Report the mean and spread across seeds.
• Compare against a scripted baseline and a single-agent baseline.
• Test with unseen partners and unseen opponents.
• Measure both team return and per-agent return.
• In competitive games, estimate exploitability.
• Stress test with noise in sensors and delays in actions.
• Log the full rollouts so that failures can be replayed.

A pre-training checklist

Before launching a long run, ask these questions:

• Is the reward aligned with what the operator truly wants?
• Does each agent have enough information to act well?
• Are the agents symmetric, so that parameter sharing is valid?
• Is the simulator fast enough to give millions of steps?
• Is there a clear stop rule and a clear success test?
• Are the logs stored in a schema that other tools can read?

Stabilizing training

• Use a replay buffer carefully. Old data was produced by older partners, so it can mislead.
• Keep learning rates small when many agents update at once.
• Normalize observations and rewards.
• Clip gradients to avoid sudden jumps.
• Update agents in turns at first if joint updates are unstable.
• Add entropy bonuses so that agents keep exploring.
• Save checkpoints often and keep the best by a fixed evaluation, not by training reward.

When not to use MARL

• When a plain optimizer solves the task with a known model.
• When the number of agents is tiny and a single controller can decide for all.
• When no simulator exists and real trials are costly or risky.
• When rules are required by regulation and must be fully explained.

Reading results with care

• A rising training curve alone does not prove the agents cooperate well.
• Watch for one agent doing all the work while the others idle.
• Look at replays of a few episodes, not only averages.
• Compare results when one agent is removed. A good team degrades gently.
• Check that behavior holds when the number of agents changes.

Common mistakes

• Ignoring non-stationarity and expecting single-agent guarantees to hold.
• Using a shared reward with many agents and no credit assignment method.
• Testing only against the same partners used in training, which hides brittleness.
• Letting one agent dominate the shared network because its data is more plentiful.
• Reporting one lucky seed as the result.
• Making the joint action space huge when factoring it per agent would work.
• Forgetting to feed a time or episode signal when the task depends on it.
• Leaking privileged information at run time that was only meant for training.
• Treating emergent cooperation as proof of robustness without stress tests.
• Skipping a simple baseline, such as a scripted policy, to see how much learning actually adds.

Frequently asked questions

What is multi-agent reinforcement learning in one sentence?

It is reinforcement learning where several agents learn at the same time in a shared environment, so each agent must adapt to others who are also adapting.

Why is multi-agent RL harder than single-agent RL?

The main reason is non-stationarity. When other agents change their behavior, the environment an agent is trying to learn keeps shifting, so the usual convergence guarantees no longer apply. The joint action space also grows quickly, and it is hard to tell which agent deserves credit for a shared outcome.

What does centralized training with decentralized execution mean?

During training the learner may use extra information, such as all agents' observations and actions, to judge the situation well. At run time each agent acts using only its own observation and its own policy. This gives stable learning without requiring a central controller in deployment.

Do agents in MARL have to cooperate?

No. Agents can be fully cooperative, fully competitive or mixed. Cooperative tasks share one reward, competitive tasks like two-player games are zero-sum, and mixed tasks give each agent its own partly aligned reward. The algorithm choice depends on which of these applies.

What is the credit assignment problem?

When many agents share one reward, each agent receives a noisy signal and cannot tell how much its own action mattered. Methods such as counterfactual baselines, difference rewards and value decomposition try to split the team reward into fair per-agent signals.

How is self-play used in multi-agent RL?

In self-play an agent trains against copies or past versions of itself. As the agent improves, so does its opponent, which creates a natural curriculum without human data. To avoid cycling between strategies, many systems keep a pool of earlier opponents and sample from it.

Can PPO be used for multiple agents?

Yes. PPO is a common base algorithm in multi-agent work, often with parameter sharing and a centralized value function. It is popular because it is stable and simple to tune, though credit assignment and partner diversity still need attention.

How does MARL relate to tabular data?

MARL produces tabular logs, with one row per agent per step, and it consumes tabular predictions, such as failure risk or demand forecasts. Tabular foundation models can learn from these logs, fill missing sensor values and flag unusual behavior, all without hand-built features.

Is MARL ready for real-world control?

It is used successfully in simulation and in some controlled applications, but real deployments need caution. Sample cost, safety, robustness to new partners and the gap between simulator and reality all limit use. Most production systems combine learned policies with rule-based safeguards.

What should I learn before studying MARL?

A good base is single-agent reinforcement learning, including Markov decision processes, value functions and policy gradients, plus some basic game theory. After that, read about independent learning, centralized critics and value decomposition.

Glossary

• Action space: the set of moves an agent can make.
• Agent: a decision maker that senses and acts.
• Centralized critic: a value estimator that sees information from all agents during training.
• Credit assignment: deciding which action caused which outcome.
• Dec-POMDP: a cooperative decision model with partial observation and a shared reward.
• Discount factor: a number below one that reduces the weight of distant rewards.
• Emergent behavior: group patterns that arise from learning rather than design.
• Equilibrium: a stable joint strategy where no agent gains by deviating alone.
• Exploitability: how much a best-response opponent can win against a policy.
• Joint action: the combined actions of all agents at one step.
• Markov game: a multi-player extension of the Markov decision process.
• Non-stationarity: change in the environment caused by other learners.
• Observation: what an agent can see of the state.
• Parameter sharing: reusing one set of weights across several agents.
• Policy: the mapping from observation to action.
• Reward shaping: adding extra signals to guide learning.
• Self-play: training against oneself or past versions.
• Value decomposition: splitting a team value into per-agent parts.
• Zero-sum game: a game where total reward is fixed and one side's gain is the other's loss.

Summary

Multi-agent RL extends reinforcement learning to settings where several agents learn together, whether they cooperate, compete or both. Its core difficulties are non-stationarity, credit assignment and a growing joint action space. Methods such as independent learning, centralized training with decentralized execution, value decomposition and self-play address these in different ways.

For tabular work, MARL both generates mixed-type logs and depends on tabular predictors for risk, demand and anomaly signals. Models that learn directly from raw tables, covered in /model/research, /model/architecture and /model/benchmarks, can sit alongside learned agents to impute gaps, flag drift and supply fast in-context predictions. Start simple, design rewards with care, test against unseen partners, and keep safeguards outside the learned policy.

Related

It belongs to Reinforcement learning in my stack, under AI and machine learning. Others in the same group: