Reinforcement learning

Overview

Reinforcement learning is a way of learning to make decisions by trial and error. An agent acts in an environment, receives rewards or penalties, and adjusts its behavior to collect more reward over time. There is no answer key that says which action was right. There is only a score that arrives after the fact, often delayed.

It suits problems where choices affect future situations: games, robot control, pricing, ad placement, inventory, and recommendation. Unlike supervised learning, which predicts a known target, reinforcement learning chooses actions. Its link to tabular foundation models is practical. Decision problems often start from logged tables of states, actions, and outcomes, and models that read tables well can help estimate rewards, fill gaps, and flag unsafe data.

What it is

Reinforcement learning, often shortened to RL, is a framework for sequential decision making. At each step the agent sees a state, picks an action, and the environment returns a reward and a new state. The agent's goal is to maximize total reward over time, not just the next reward.

Picture teaching a dog a trick. You do not give the dog a manual. You give a treat when it does something close to what you want. Over many tries the dog links its actions to treats. RL works the same way, but with numbers and code.

The main parts

• Agent: the learner and decision maker.
• Environment: everything the agent interacts with.
• State: a description of the current situation.
• Action: a choice the agent can make.
• Reward: a number that says how good the last step was.
• Policy: the rule the agent uses to pick actions.
• Value function: an estimate of how much reward to expect from a state or a state and action pair.
• Model of the environment: an optional guess of how states change.

What makes it different

• Feedback is evaluative, not instructive. It tells you how good an action was, not what the best action was.
• Feedback is delayed. A move now may pay off many steps later.
• Data depends on the agent. The actions it takes decide what it sees.
• It must balance exploration, trying new things, with exploitation, using what works.

Close relatives

• Supervised learning learns from correct answers.
• Unsupervised learning finds structure without answers.
• Active learning also chooses what to look at, but to get labels, not rewards.
• Meta-learning can be applied to RL so that agents adapt quickly to new tasks.
• Federated learning can train RL agents across devices without sharing raw data.

Where it came from

RL has two old roots. One is the psychology of animal learning. Early twentieth century work on the law of effect said that actions followed by good outcomes become more likely. The other is optimal control and dynamic programming, developed in the 1950s, which gave tools to compute the best decisions when the environment is known.

Markov decision processes became the standard math model. In the 1980s, temporal difference learning combined ideas from both roots. It lets an agent learn from the difference between successive predictions, rather than waiting for a final outcome. Q-learning, introduced at the end of the 1980s, learned the value of actions without a model of the environment.

A well known backgammon program in the early 1990s showed that self-play and temporal difference learning could reach expert level. Policy gradient methods, which tune the policy directly, grew through the 1990s. Standard textbooks in the late 1990s and later editions helped define the field.

In the 2010s, deep reinforcement learning joined neural networks with RL. Agents learned to play video games from raw pixels and to beat top players in board games through self-play. Actor-critic methods, trust region methods, and proximal policy optimization became common tools. More recently, RL from human feedback has been used to tune language models. The research overview shows how these ideas relate to table-based models.

How it works

The core loop is simple. The details differ by algorithm.

The interaction loop

1. The agent observes the current state.
2. It picks an action using its policy.
3. The environment responds with a reward and a new state.
4. The agent updates its estimates or policy.
5. The loop repeats until the episode ends or forever.

The goal: discounted return

The agent wants to maximize the sum of future rewards. Future rewards are usually discounted by a factor gamma between 0 and 1. A reward one step away counts fully, one two steps away counts gamma, and so on. Think of money: a dollar today is worth more than a dollar next year. A gamma near 1 makes the agent patient. A gamma near 0 makes it short-sighted.

Value functions

The value of a state is the expected total reward from that state when following a policy. The value of a state and action pair, called a Q-value, is the expected reward from taking that action, then following the policy. If you know Q-values, the best policy is to pick the action with the highest Q.

Bellman equation

The Bellman idea says the value of a situation equals the immediate reward plus the discounted value of where you land. It is like estimating a trip time: time to the next town plus time from there. Most RL algorithms are ways to make estimates satisfy this relation.

Q-learning

Q-learning keeps a table of Q-values, one per state and action. After each step it nudges the Q-value toward a better estimate.

• New estimate: reward plus gamma times the best Q-value in the next state.
• Update: move the old Q a small step, set by a learning rate, toward the new estimate.

It is off-policy, which means it can learn the best policy while following a different, more exploratory one.

Exploration

An agent that always takes the best known action may miss better options. Common strategies:

• Epsilon-greedy: with small probability pick a random action.
• Softmax: pick actions with probability tied to their value.
• Optimism: start with high value guesses so every action gets tried.
• Upper confidence bounds: favor actions with uncertain but promising values.
• Thompson sampling: sample a plausible world and act as if it were true.

Policy gradients

Instead of learning values, tune the policy directly. If an action led to more reward than expected, make it more likely. If less, make it less likely. Policy gradient methods work well with continuous actions, but they have high variance.

Actor-critic

Two parts work together. The actor picks actions. The critic estimates values and tells the actor how well it did. The critic reduces the noise in the actor's learning signal.

Deep reinforcement learning

When states are large, a table of Q-values is impossible. A neural network approximates the value or the policy instead. Tricks such as replay buffers, which reuse past experience, and target networks, which keep targets steady, help training stay stable.

Model-based versus model-free

• Model-free methods learn values or policies directly from experience.
• Model-based methods learn a model of how the environment changes and plan with it.

Model-based methods use data well but suffer if the model is wrong. Model-free methods are simpler but need more experience.

Bandits: the simplest case

A multi-armed bandit has one state and many actions, like a row of slot machines. There are no long-term effects, so the problem is only about balancing exploration and exploitation. Contextual bandits add features describing the situation, which makes them close to supervised learning on tables with partial feedback. They power many recommendation and pricing systems.

Offline reinforcement learning

Sometimes you cannot let an agent experiment. You only have a log of past decisions and outcomes. Offline RL learns a policy from that fixed log. This is hard, because the log covers only the actions that were taken, so the value of untried actions must be guessed. Good offline methods stay close to the logged behavior unless evidence supports a change.

Families of RL algorithms at a glance

On-policy versus off-policy in daily terms

On-policy learning is like a cook who improves by tasting only their own dishes. Off-policy learning is like a cook who can also learn from watching other cooks. Off-policy methods reuse old data better, but they need care when the old cooks behaved very differently from the current one.

Reward shaping in plain words

If a reward arrives only at the very end, the agent has little to learn from. Shaping adds small hints along the way, like checkpoints in a long race. A well shaped reward speeds learning. A badly shaped one teaches the agent to chase the checkpoints instead of finishing the race.

Key concepts and terms

Markov decision process

A Markov decision process, or MDP, is the formal setting for RL. It has states, actions, transition probabilities, rewards, and a discount factor. The Markov property says the present state holds all the information needed to decide. If the agent cannot see the full state, the problem is a partially observable MDP, and the agent may need memory.

A worked example

This example trains a tabular Q-learning agent on a tiny corridor. The agent starts at the left end of five cells and must reach the right end for a reward.

import numpy as np

n_states = 5          # cells 0..4, goal at 4
actions = [-1, +1]    # move left or right
Q = np.zeros((n_states, len(actions)))

alpha, gamma, epsilon = 0.1, 0.9, 0.2
rng = np.random.default_rng(0)

def step(state, a_idx):
    nxt = min(max(state + actions[a_idx], 0), n_states - 1)
    reward = 1.0 if nxt == n_states - 1 else -0.01
    done = nxt == n_states - 1
    return nxt, reward, done

for episode in range(500):
    s = 0
    done = False
    while not done:
        if rng.random() < epsilon:
            a = int(rng.integers(len(actions)))
        else:
            a = int(np.argmax(Q[s]))
        s2, r, done = step(s, a)
        target = r + (0 if done else gamma * Q[s2].max())
        Q[s, a] += alpha * (target - Q[s, a])
        s = s2

print(np.round(Q, 2))
print("policy:", ["R" if Q[i].argmax() == 1 else "L" for i in range(n_states)])

Reading the example

• The table Q has one row per cell and one column per action.
• A small penalty per step pushes the agent to reach the goal quickly.
• Epsilon-greedy choice mixes random moves with the best known move.
• After training, the policy should say move right in every cell.
• Q-values rise toward the goal, since cells closer to the goal have less discounting.

A numeric walk-through of one update

Say the agent is in cell 3 and moves right to cell 4, the goal, and receives a reward of 1.0. The episode ends, so the target is simply 1.0. If Q for cell 3 and action right was 0.0, and alpha is 0.1, the update gives 0.0 plus 0.1 times (1.0 minus 0.0), which is 0.1.

Later, in cell 2, the agent moves right to cell 3 with reward minus 0.01. The best Q in cell 3 is now 0.1. The target is minus 0.01 plus 0.9 times 0.1, which is 0.08. The old Q was 0, so the new Q is 0.1 times 0.08, about 0.008. Value is slowly flowing backward from the goal, one step at a time. After many episodes, every cell has a sensible value.

A bandit example in words

Suppose a site has three banners. Banner A shows a click rate of 2 percent on 1,000 views, banner B shows 3 percent on 50 views, and banner C has not been shown. An epsilon-greedy rule mostly shows A, but sometimes tries B and C. Thompson sampling would draw a plausible click rate for each, using uncertainty, and show whichever draw is highest. B and C get tried more because they are less certain. As evidence grows, the best banner wins most of the traffic.

A second numeric walk-through: discounted return

An agent receives rewards of 1, 0, 0, and 10 over four steps. With gamma equal to 0.9, the return from the first step is 1 plus 0.9 times 0, plus 0.81 times 0, plus 0.729 times 10. That is 1 plus 7.29, or 8.29. With gamma equal to 0.5, the return is 1 plus 0.125 times 10, or 2.25. The same history is worth much less to a short-sighted agent. Choosing gamma is therefore a statement about how much the future matters.

Applications on decision logs

• Pricing: choose a price for each context to maximize profit over time.
• Recommendation: choose what to show next, learning from clicks and purchases.
• Inventory: choose order quantities given stock, demand, and lead times.
• Maintenance: choose when to service a machine given its sensor state.
• Treatment planning: choose a sequence of steps given a patient history, with strict safety review.
• Resource scheduling: choose how to assign jobs to machines.
• Energy: choose when to charge or discharge storage.

Each of these problems can be stored as a table of states, actions, and results.

Strengths and limitations

Strengths

• It handles sequential decisions where today's action changes tomorrow's options.
• It needs no labeled answers. A reward signal is enough.
• It can find strategies that humans did not think of.
• It adapts as the environment changes, if it keeps learning.
• It optimizes the real goal, such as long-term revenue, rather than a proxy.

Limitations

• It can be very sample hungry. Many tasks need millions of steps.
• Reward design is hard. Agents exploit loopholes in a badly written reward.
• Training can be unstable and sensitive to settings and random seeds.
• Real-world exploration can be costly or unsafe.
• Credit assignment is hard when rewards come late.
• Results are hard to explain and to reproduce.
• Offline data may not cover the actions that matter.
• Simulators differ from reality, so policies may not transfer.

Typical failure patterns

How it compares with alternatives

Supervised learning plus a decision rule

Often a simpler design works. Train a supervised model to predict outcomes, then pick the action with the best predicted outcome. This ignores long-term effects, but it is easy to build, test, and explain. RL earns its place when actions change the future state of the system, or when feedback is partial and biased by past choices.

Role in tabular foundation models

RL and tabular foundation models meet in the data and in the decision step. Many decision systems run on tables.

Logged decisions are tables

A log of decisions is a table. Each row has a state described by columns, an action taken, a reward observed, and often the next state. Offline RL and contextual bandits learn from exactly such tables. Any model that reads raw tables well can therefore be applied here.

Reward and outcome prediction

A common sub-task is predicting the reward for a given state and action. That is a supervised prediction over a table, with mixed types and missing values. A pre-trained table model, such as TabPFN, can give reward estimates from few logged rows with no long training run. This helps when data are scarce, as in early A/B tests.

Row and column structure

States are rows. Features of the state are columns. The order of columns carries no meaning, and neither does the order of rows within a batch. A model built with attention across rows and columns respects these symmetries. See the architecture page and transformers.

Mixed types and missing values

Real decision logs mix counts, categories, times, and free flags. Many fields are missing because sensors fail or users skip questions. A model that learns during pre-training how to treat missing cells can fill them and use missingness as a signal. This links to imputation networks.

No column names

If a logged table has anonymous columns, a foundation model can still infer the role of each from its distribution and relations. That helps when reusing the same decision model across many sources.

Pre-training on synthetic and real tables

Table models are pre-trained on many generated tables, built with synthetic data generation. A similar idea exists in RL: simulators produce synthetic experience to train agents. Simulated tables can also be used to test policies before they touch real customers.

In-context learning for decisions

With a table model, a handful of logged outcomes can be placed in context, and the model predicts the outcome of a new action. This acts like a fast surrogate for the environment. It can support planning, ranking of candidate actions, or off-policy evaluation, where you estimate how a new policy would have done on old data.

How a table model can sit inside an RL system

1. Logged rows feed a table model that predicts reward for each candidate action.
2. A decision rule picks the action with the best predicted reward, plus some exploration.
3. New outcomes are added to the log as new rows.
4. The table model sees the larger table as context the next time, with no retraining step.
5. A monitor compares new rows with older rows and flags drift.

This pattern is a contextual bandit with a pre-trained predictor. It trades long-horizon planning for simplicity and speed.

Continual learning

Environments drift. A decision system must keep updating without forgetting. Continual learning in RL and in table models share goals: adapt to new patterns, protect old skills, and detect when the world has changed.

Anomaly detection and safety

A table model that learns what normal states look like can flag unusual states. This supports anomaly detection as a safety layer: if the current state is far from the data the policy was trained on, fall back to a safe default or ask a human.

Sector and semantic context

Knowing that a table describes, say, a logistics operation or a retail store helps choose reward definitions and constraints. Semantic classification of tables and columns by a foundation model can feed that choice. Benchmarks for these models are discussed on the benchmarks page.

Off-policy evaluation in brief

Suppose a store used a random discount rule for a month and logged each offer with its probability. You can estimate how a new rule would have performed.

• Inverse propensity weighting re-weights logged rows by how likely the new rule was to pick the same action.
• Direct methods train a model of reward and use it to score the new rule.
• Doubly robust methods combine both and stay accurate if either part is good.

A reward model trained on a table is the core of the direct method. This is where a strong table predictor helps most.

Why logging probabilities matters

If you do not store the chance that each action was chosen, you cannot correct for the bias of the old policy. The data then look as if the old rule were the only sensible one, and the new policy cannot be judged fairly. Adding one extra column for the action probability costs almost nothing and saves the whole evaluation.

Practical guidance and best practices

Decide if RL is the right tool

• Do actions change future states? If not, a supervised model with a decision rule may do.
• Is there a clear, measurable reward? If not, define one first.
• Can you explore safely? If not, use simulation or offline data.
• Is there enough data or a good simulator?

Design the problem

• Write the state so it contains what the agent needs and nothing that leaks the future.
• Keep the action set small and meaningful.
• Define the reward to match the real goal. Add penalties for harmful behavior.
• Check for loopholes by asking how a clever agent could score high without doing the job.
• Add constraints for safety, cost, and fairness.

Train with care

• Start with a simple baseline policy, such as the current business rule.
• Try a bandit or a one-step model before full RL.
• Use several random seeds and report the spread.
• Log every state, action, reward, and the probability of the chosen action. You need these for later evaluation.
• Keep a replay of past data for off-policy checks.

Safe exploration patterns

• Start with the current policy and let the agent deviate only a little.
• Use a cap on how much any single decision can change.
• Test in a simulator or a small slice of traffic first.
• Add a monitor that stops exploration when a key metric drops.
• Keep a human override for high stakes actions.

Reward design patterns

Evaluate before deploying

• Use off-policy evaluation on logged data to estimate the new policy.
• Run a small, controlled online test with guard rails.
• Compare against the current policy, not only against random.
• Monitor reward, constraint violations, and drift after launch.

A short checklist

• Is the reward aligned with the real goal?
• Is exploration safe?
• Are logs complete, including action probabilities?
• Is there a fallback policy?
• Are results stable across seeds?
• Is there a human review path?

Common mistakes

• Using RL when a supervised model and a simple rule would do.
• Writing a reward that can be gamed.
• Ignoring safety during exploration.
• Evaluating on the same environment seeds used for training.
• Reporting a single lucky run.
• Not logging action probabilities, which makes later off-policy evaluation impossible.
• Letting future information leak into the state.
• Using too high a discount factor on noisy rewards, so errors pile up.
• Training offline on data that never tried the actions the new policy wants.
• Deploying without a fallback or monitoring.
• Forgetting that the environment drifts, so a policy trained once will age.

Frequently asked questions

What is reinforcement learning in simple words?

Reinforcement learning is learning by trying things and seeing what happens. An agent takes actions, gets a reward or penalty, and slowly learns which actions lead to the best results over time. It is like training a pet with treats, but done with numbers.

How is reinforcement learning different from supervised learning?

Supervised learning is given the correct answer for each example. Reinforcement learning only gets a reward after acting, and the reward may arrive late. The agent must also decide what to try, which changes the data it sees. This makes RL harder but suited to decision problems.

What is the exploration versus exploitation trade-off?

Exploitation means using the best option known so far. Exploration means trying other options that might turn out better. An agent that never explores can get stuck with a mediocre choice, and one that only explores never profits from what it learned. Good algorithms balance both.

What is Q-learning?

Q-learning is a method that learns the value of each action in each state. After every step it nudges its estimate toward the reward received plus the discounted best value of the next state. Over time, picking the action with the highest value gives a good policy.

What are multi-armed bandits?

A multi-armed bandit is a simplified RL problem with one state and several actions, like slot machines with unknown payouts. The task is to find the best action while still earning reward. Contextual bandits add features describing each situation, which makes them common in recommendation and pricing.

Can reinforcement learning work with tabular data?

Yes. Many decision problems are stored as tables of states, actions, and rewards. Contextual bandits and offline RL learn from these logs. Table-focused models can predict rewards, fill missing values, and flag unusual states, which supports the decision step.

What is offline reinforcement learning?

Offline RL learns a policy from a fixed log of past decisions without further interaction. It is useful when experiments are costly or risky. The main difficulty is that the log covers only actions that were taken, so the value of other actions must be estimated with care.

Why is reward design so hard?

An agent optimizes exactly what the reward says, not what you meant. If the reward has a loophole, the agent will find it. Good reward design needs careful thought about side effects, constraints, and long-term goals, plus testing for unintended behavior.

How does RL relate to tabular foundation models?

Decision logs are tables, and many sub-tasks in RL, such as predicting rewards or filling missing state values, are table prediction tasks. A pre-trained table model can provide these estimates with little data. It can also act as a safety check by flagging states far from the training data.

How much data does reinforcement learning need?

It depends on the problem. Simple bandit problems can work with hundreds or thousands of interactions. Deep RL in complex settings can need millions of steps. Simulators, model-based methods, and pre-trained models reduce the amount of real experience needed.

Glossary

• Action: A choice the agent can make.
• Actor-critic: A method with a policy part and a value part that guide each other.
• Agent: The learner that acts.
• Bandit: A decision problem with one state and several actions.
• Bellman equation: A relation tying a state's value to its immediate reward and the next state's value.
• Contextual bandit: A bandit where features describe each situation.
• Discount factor: A number below one that reduces the weight of future reward.
• Environment: The world in which the agent acts.
• Episode: One run from start to finish.
• Epsilon-greedy: Choosing a random action with small probability and the best action otherwise.
• Exploration: Trying options to gain information.
• Markov decision process: The formal model of states, actions, transitions, and rewards.
• Off-policy: Learning about one policy while following another.
• Offline RL: Learning from a fixed log of past experience.
• Policy: The rule that maps states to actions.
• Q-value: Expected return of an action in a state.
• Replay buffer: A store of past experience for reuse in training.
• Return: Total discounted reward.
• Reward: The feedback number after an action.
• State: A description of the current situation.
• Value function: Expected return from a state.

Summary

Reinforcement learning teaches an agent to act by rewarding good outcomes. Its building blocks are states, actions, rewards, policies, and value functions. Methods range from Q-learning on small tables to deep actor-critic systems, with bandits as the simplest case and offline RL for settings where live experiments are not possible.

Success depends less on the algorithm than on the setup: a reward that matches the real goal, safe exploration, honest evaluation, and a fallback plan. Many problems are better solved by supervised prediction plus a simple decision rule.

For tabular foundation models, the link is the data. Decision logs are tables with mixed types, missing values, and sometimes no useful column names. Table models can estimate rewards in context, impute missing state values, flag unusual states, and help with off-policy checks. See the research, architecture, and benchmarks pages for how such models are built and measured.

Related

It belongs to Learning paradigms in my stack, under AI and machine learning. Others in the same group: