DQN

Overview

DQN, short for deep Q-network, is a reinforcement-learning algorithm that uses a neural network to estimate how good each action is in each situation. The network takes a state as input and returns one number per possible action, called the Q-value. The agent then picks the action with the highest value, most of the time. DQN made it practical to combine Q-learning with deep networks, and it became famous for learning to play many video games from raw pixels.

Two ideas make it stable: experience replay, which stores past transitions and trains on random samples of them, and a target network, which is a slowly updated copy of the main network used to compute training targets. DQN works with a discrete set of actions, such as "go left", "go right" or "assign job A to machine 2". It sits in the wider family of reinforcement learning methods and has direct descendants such as Double DQN, Dueling DQN and Rainbow.

What it is

Reinforcement learning describes an agent that interacts with an environment over time. At each step the agent sees a state, picks an action, receives a reward and moves to a new state. The goal is to choose actions that maximize the total future reward.

DQN is a value-based method. It does not learn a policy directly. It learns the action-value function, written Q(s, a), which is the expected total future reward if the agent takes action a in state s and then behaves well afterwards. Once Q is known, the best action in any state is simply the one with the highest Q.

An everyday analogy: imagine choosing a route home. For each junction and each possible turn, you keep a mental score for how quickly it usually gets you home. After many trips your scores become good, and at each junction you take the turn with the best score. DQN does the same, using a neural network as a very large, flexible scoreboard that can handle junctions it has never seen exactly.

The discount factor

Future rewards are usually worth a bit less than immediate ones. A discount factor γ between 0 and 1 scales them. With γ = 0.99, a reward 100 steps away counts for about a third of the same reward now. The discount keeps the sums finite and expresses a preference for sooner rewards.

What DQN needs from the problem

• A discrete action set. The network outputs one value per action. For continuous actions, other methods such as DDPG, TD3 or SAC are used.
• A state that carries enough information. Ideally the state satisfies the Markov property: the current state holds what is needed to decide, without the full history. Frame stacking and recurrent layers help when it does not.
• A reward signal. It can be sparse or noisy, but it must exist and be linked to the goal.
• Lots of interaction data. DQN is sample hungry. It usually runs against a simulator or a large log.

Where it came from

Q-learning is an older idea. It was introduced in 1989 as a way to learn action values from experience, without needing a model of the environment. For years it was used with tables of values, one entry per state and action, which only works for small problems. Attempts to replace the table with a function approximator, such as a neural network, often diverged, because three things together are risky: function approximation, bootstrapping (updating a guess from another guess) and off-policy learning. This is sometimes called the deadly triad.

Early successes with neural networks in reinforcement learning, such as a backgammon-playing system in the 1990s, were hard to repeat on other problems. "Neural fitted Q iteration" in the mid 2000s took steps toward stable batch learning.

The breakthrough was "Playing Atari with Deep Reinforcement Learning" (2013), which showed a convolutional network learning control policies from raw pixels. A longer paper, "Human-level control through deep reinforcement learning" (2015), added the target network and ran the method across a large set of classic Atari games with one architecture and one set of hyperparameters. Many games reached or passed human-level scores, and the work set off a wave of deep reinforcement-learning research.

Improvements followed quickly: Double DQN (2015) reduced overestimation, prioritized experience replay (2015) sampled important transitions more often, Dueling DQN (2015) split value and advantage, noisy networks and distributional methods added exploration and richer targets, and Rainbow (2017) combined several of these. Policy-gradient and actor-critic methods such as PPO, TRPO and REINFORCE became popular for other settings, but DQN remains the standard teaching example and a strong baseline for discrete actions.

How it works

Step 1: the Q-learning update

Plain Q-learning updates one value at a time. After taking action a in state s, getting reward r and landing in state s', the update is:

Q(s, a) ← Q(s, a) + α × [ r + γ × max over a' of Q(s', a') - Q(s, a) ]

The term in brackets is the temporal-difference error. It compares what happened (the reward plus the best estimated future value) with what the table predicted. If the two disagree, the entry moves a little toward the new estimate. This is learning from a guess about the next step, which is called bootstrapping.

Step 2: replace the table with a network

DQN replaces the table with a neural network Q(s, a; θ) with weights θ. Training minimizes the squared difference between the network's prediction and a target:

target = r + γ × max over a' of Q_target(s', a'; θ⁻)

loss = ( target - Q(s, a; θ) )²

Here θ⁻ are the weights of the target network. In practice the Huber loss is common, because it limits the effect of large errors.

Step 3: experience replay

The agent stores each transition (s, a, r, s', done) in a replay buffer, a large queue of recent experience. At each training step it samples a random mini-batch from the buffer.

Why this helps:

• It breaks correlation. Consecutive steps in a game or process are very similar. Training on them in order makes the network chase the latest situation. Random sampling mixes old and new, like shuffling a deck of flash cards.
• It reuses data. Each transition can be used many times, which saves costly interaction.
• It smooths the data distribution. The network sees an average over many past behaviours instead of a sudden change.

See experience replay for a deeper look.

Step 4: the target network

If the same network produces both the prediction and the target, every update shifts the target as well. It is like trying to hit a moving bullseye that you are also pushing. DQN keeps a second copy of the network, the target network, whose weights are frozen for a while. Targets are computed from it. Every fixed number of steps, the main weights are copied over, or the target weights are moved slowly toward the main weights (a soft update). This makes the learning problem look like ordinary supervised regression for a stretch of time.

Step 5: exploration with epsilon-greedy

An agent that always picks the best-known action never finds out whether something else is better. DQN uses epsilon-greedy exploration. With probability ε it picks a random action, and otherwise it picks the action with the highest Q-value. ε starts high, often near 1, and decays toward a small value, such as 0.01 to 0.1, as the agent learns. Think of trying new restaurants often when you move to a new city, and sticking to favourites once you know the area.

Step 6: the full loop

1. Initialize the main network and a copy as the target network. Fill the replay buffer with random experience.
2. Observe the state.
3. Choose an action by epsilon-greedy.
4. Act, receive the reward and the next state, and store the transition.
5. Sample a mini-batch from the buffer.
6. Compute targets with the target network, and update the main network by gradient descent on the loss.
7. Every so often, update the target network.
8. Repeat until the performance stops improving.

Practical details from the original recipe

• Frame stacking. For games, the last four frames are stacked, so the network can infer motion.
• Reward clipping. Rewards are clipped to a small range, so that one learning rate works across games with different scales.
• Gradient clipping or Huber loss. This stops rare big errors from destroying the weights.
• Frame skipping. The agent acts every few frames to save compute.
• Preprocessing. Frames are resized and converted to grayscale.
• Optimizer. RMSProp or Adam, with small learning rates.

For tabular states, none of the image-specific parts apply. The network is simply a multilayer perceptron over a vector of features.

Extensions in a nutshell

Each is optional. A sensible path is to run plain DQN as a baseline, add the double target, then add the others one at a time and measure each.

Why the deadly triad matters

Three ingredients are each fine alone. Function approximation lets a network generalize. Bootstrapping lets learning happen before an episode ends. Off-policy learning lets old data be reused. Together they can make the value estimates spiral. A small error in one state spreads to similar states through the shared weights, and bootstrapping then feeds that error back into the targets. The target network, replay and small learning rates are practical ways to damp the spiral. They do not remove the risk, which is why monitoring Q-values is part of good practice.

A debugging checklist

When DQN does not learn, check these in order.

1. Environment sanity. Can a hand-written policy solve it? Does the reward move in the right direction when it should?
2. Observation sanity. Print a few states. Are scales sensible? Is anything from the future included?
3. Shapes and indexing. Confirm that the Q-value picked by gather belongs to the action taken.
4. Terminal handling. Confirm that the future term is zero at the end of an episode.
5. Target network. Confirm it is a separate copy and is updated on schedule.
6. Exploration. Confirm epsilon decays and the agent really tries all actions early on.
7. Replay. Confirm the buffer is filled before updates begin and is sampled uniformly.
8. Learning rate. Lower it by a factor of ten and see if the curve smooths.
9. Overfitting a tiny case. Train on a very small environment. If it fails there, the code has a bug.
10. Seeds. Run three seeds before concluding anything.

Beyond games: where discrete-action decisions appear

• Dispatching and scheduling. Which job goes to which resource next. See logistics for the wider domain.
• Inventory and replenishment. Order, hold or discount, with a limited set of order sizes.
• Maintenance decisions. Inspect, repair or wait, given condition data.
• Recommendation and ranking. Choose one of a set of items to show, with long-term engagement as the reward.
• Network and resource control. Switch routes or allocate capacity among a few options.
• Customer treatment. Choose among a few actions, such as a reminder, an offer or no contact, to maximize long-term value.

In all of these the value of an action shows up later, and the logs live in tables. That is the bridge to tabular work.

Key concepts and terms

A worked example

A numeric walk-through of one update

Say the agent is in state s and takes action a. It gets reward r = 1 and moves to s'. The target network gives these values for the next state.

Take γ = 0.9. The target is:

1 + 0.9 × 3.5 = 4.15

Suppose the main network currently predicts Q(s, a) = 3.0. The temporal-difference error is 4.15 - 3.0 = 1.15. The squared loss is about 1.32. A gradient step lowers the prediction error by nudging the weights so that Q(s, a) moves toward 4.15. If the episode had ended at s', the target would have been just r = 1, because there is no future to add.

A small code example

The code below trains a DQN on a toy dispatching problem: choose one of three machines for each arriving job. The environment is a stand-in; the structure is what matters.

import random
from collections import deque
import numpy as np
import torch
from torch import nn

class ToyDispatchEnv:
    """State: queue lengths of 3 machines plus job size. Action: pick a machine."""
    def reset(self):
        self.q = np.zeros(3, dtype="float32")
        self.t = 0
        return self._obs()
    def _obs(self):
        self.job = np.random.uniform(0.1, 1.0)
        return np.append(self.q, self.job).astype("float32")
    def step(self, a):
        self.q[a] += self.job
        self.q = np.maximum(self.q - 0.4, 0)           # machines work off load
        reward = -float(self.q.max())                   # penalize the longest queue
        self.t += 1
        return self._obs(), reward, self.t >= 200

def mlp(n_in, n_out):
    return nn.Sequential(nn.Linear(n_in, 64), nn.ReLU(),
                         nn.Linear(64, 64), nn.ReLU(),
                         nn.Linear(64, n_out))

env = ToyDispatchEnv()
q_net, tgt_net = mlp(4, 3), mlp(4, 3)
tgt_net.load_state_dict(q_net.state_dict())
opt = torch.optim.Adam(q_net.parameters(), lr=1e-3)
buffer = deque(maxlen=50_000)
gamma, eps, eps_min, eps_decay = 0.95, 1.0, 0.05, 0.999
batch_size, sync_every, step_count = 64, 500, 0

for episode in range(300):
    s, done = env.reset(), False
    while not done:
        if random.random() < eps:
            a = random.randrange(3)
        else:
            with torch.no_grad():
                a = int(q_net(torch.from_numpy(s)).argmax())
        s2, r, done = env.step(a)
        buffer.append((s, a, r, s2, float(done)))
        s = s2
        eps = max(eps_min, eps * eps_decay)
        step_count += 1

        if len(buffer) >= 1000:
            batch = random.sample(buffer, batch_size)
            bs, ba, br, bs2, bd = map(np.array, zip(*batch))
            bs, bs2 = torch.from_numpy(bs), torch.from_numpy(bs2)
            ba = torch.from_numpy(ba).long().unsqueeze(1)
            br, bd = torch.from_numpy(br).float(), torch.from_numpy(bd).float()
            with torch.no_grad():
                nxt = tgt_net(bs2).max(dim=1).values
                target = br + gamma * nxt * (1 - bd)
            pred = q_net(bs).gather(1, ba).squeeze(1)
            loss = nn.functional.smooth_l1_loss(pred, target)
            opt.zero_grad(); loss.backward()
            nn.utils.clip_grad_norm_(q_net.parameters(), 10.0)
            opt.step()

        if step_count % sync_every == 0:
            tgt_net.load_state_dict(q_net.state_dict())

Reading the code

• mlp builds the Q-network: four input features, three outputs, one per machine.
• The deque is the replay buffer. Old transitions fall off the end.
• Epsilon-greedy picks random machines early and the best machine later.
• The target is reward + gamma × max next Q, computed with the frozen target network, and set to the plain reward when the episode is done.
• gather picks the predicted Q-value of the action actually taken.
• The Huber loss (smooth_l1_loss) and gradient clipping keep updates tame.
• Every 500 steps the target network is synchronized.

What to watch while it trains

• Average episode reward over time, as a moving average.
• The size of the loss. It may rise as the agent explores new situations, which is normal.
• The average predicted Q-value. If it grows without bound while rewards do not, values are being overestimated. See Double DQN.
• The fraction of random actions, as epsilon decays.
• Variation across random seeds. Reinforcement-learning results are noisy, so use several seeds.

Strengths and limitations

Strengths

• Conceptually simple and well understood.
• Off-policy, so it can learn from stored data from earlier behaviour.
• Sample reuse through replay makes better use of costly interaction.
• Works directly from high-dimensional input, such as images, or from plain feature vectors.
• Many well-tested extensions and open implementations exist.
• A good first choice for problems with a small to moderate discrete action set.

Limitations

• Discrete actions only. Continuous control needs other methods or discretization.
• Overestimation bias. Taking a maximum over noisy estimates inflates values. Double DQN addresses this.
• Sample inefficiency. It often needs millions of steps.
• Instability. The deadly triad can cause divergence. Results depend on hyperparameters and seeds.
• Sparse rewards. Epsilon-greedy exploration struggles when rewards are rare.
• Partial observability. Without memory, it fails when the state hides important history.
• Large action sets. The output layer grows with the number of actions, and many actions are rarely tried.
• Offline use is risky. Learning only from a fixed log can lead to values for actions the log never contained.
• Hard to explain. Q-values are a black box without extra analysis.

How it compares with alternatives

If actions are discrete and you have a simulator or lots of logged transitions, DQN and its descendants are a natural first try. If the action space is continuous, move to actor-critic methods. If you cannot interact with the environment at all, consider offline reinforcement learning or imitation first.

Role in tabular foundation models

DQN is not a table model. It learns decisions, while tabular foundation models learn predictions from rows and columns. The two meet in several concrete ways, and it is worth being clear about each.

The state is often a row

In business and industrial settings, the state an agent sees is frequently a record: machine readings, queue lengths, stock levels, time of day, order attributes. That is a table row. A DQN input layer over such a vector treats columns as plain numbers. This leaves out the things tabular models handle well: mixed types, missing values, categorical codes and unnamed columns. A tabular encoder can turn each row into an embedding first, and the Q-network then works on the embedding. Pre-trained table encoders of the kind described on the architecture page are natural candidates, because they already cope with mixed types and gaps.

Logged tables as training data

Many decision problems already have years of logged decisions in tables: which job went to which machine, what the queue looked like, what happened next. Because DQN is off-policy, it can in principle learn from such logs. This is offline reinforcement learning, and it needs care. The logged policy covers only some actions, so the learned Q-values can be too optimistic for actions that were rarely taken. Conservative variants and behaviour constraints are used to limit this. Good data hygiene on the log, such as consistent timestamps, complete transitions and correct rewards, matters more than the choice of algorithm.

Missing values and imputation in the state

Sensors drop out, and fields arrive late. A DQN that sees a zero for a missing reading learns the wrong thing. Using a model trained for imputation, or feeding an explicit mask along with values, keeps the state honest. Training the encoder with random masks, as in masked cell modeling, makes it robust to the same gaps at decision time.

Simulators from tables

DQN needs many interactions, which are often impossible on real equipment. A common answer is a simulator or a digital twin fitted to historical tables. A tabular predictor can serve as the transition model: given a state and an action, predict the next state and reward. Synthetic generation, as in synthetic data generation, can extend rare situations. The caution is model error: a policy that exploits the simulator's flaws will fail in the real world.

Value estimation as a tabular regression problem

At its core, fitting Q(s, a) is regression on a table of transitions: features are the state and action, the label is the bootstrapped target. In principle, any tabular regressor can play the role of the network, including tree ensembles such as XGBoost, as in fitted Q iteration. In-context tabular models could also be used to estimate values from a batch of logged transitions without training a new network, although using them in a bootstrapped loop is an open research area and should be tested carefully, not assumed.

Anomaly detection and safety

Safe operation requires knowing when the state is unlike anything seen in training. Anomaly detection over the state embedding can warn when the agent is out of its depth, and a fallback rule can take over. This is useful in industrial control, where predictive maintenance and Industry 4.0 systems collect rich tabular telemetry.

Continual learning and drift

Environments change: new machines, new demand patterns, new products. A DQN trained once will drift out of date. Continual learning methods such as elastic weight consolidation help the network adapt without forgetting. Monitoring the state distribution links to domain adaptation.

Sector and semantic context

Where decisions depend on what a table column means, such as whether a field is a price, a quantity or a duration, semantic labelling from semantic sector heads can help build correct state features automatically. This is especially useful when the same agent design is rolled out across many customers or sites, each with its own schema.

An honest summary of the relationship

Tabular foundation models and DQN do different jobs. The first predicts and represents, the second decides. The foundation model can supply state encodings, imputed values, anomaly flags and simulators. DQN supplies the decision layer on top. See the research and benchmarks pages for the prediction side of this picture.

Practical guidance and best practices

1. Start with a clear problem statement. Write down the states, the discrete actions, the reward and the episode length before any code.
2. Design the reward with care. The agent will exploit loopholes. Test the reward with simple hand-written policies first.
3. Build or validate a simulator. Compare its transitions with logged data before trusting results.
4. Normalize inputs. Scale features to similar ranges. Large raw values make learning unstable.
5. Use a replay buffer large enough to be diverse, often tens of thousands to millions of transitions.
6. Warm up the buffer with random or heuristic behaviour before the first updates.
7. Update the target network slowly. Hard copies every few thousand steps or a soft update with a small rate both work.
8. Use the Huber loss and gradient clipping.
9. Try Double DQN first. It is a one-line change and often helps.
10. Run several seeds. Report means and spreads, not single runs.
11. Compare with simple baselines. Rules, greedy heuristics and a random policy give context to the scores.
12. Evaluate with epsilon near zero, on separate evaluation episodes, and track the actions chosen.
13. Add safety limits. Constrain actions that would break rules, and fall back to a safe policy on unusual states.
14. Log everything. Seeds, hyperparameters, environment versions and checkpoints. A tracking tool such as MLflow helps.

Typical hyperparameters

A small configuration sketch

dqn:
  env: dispatch_sim_v1
  state_features: [queue_1, queue_2, queue_3, job_size, hour_of_day]
  normalize: standard
  network:
    hidden: [128, 128]
  gamma: 0.97
  optimizer:
    name: adam
    lr: 5.0e-4
  replay:
    capacity: 200000
    warmup_steps: 5000
  target:
    update: soft
    tau: 0.005
  exploration:
    start: 1.0
    end: 0.05
    decay_steps: 100000
  evaluation:
    every_steps: 10000
    episodes: 20
    seeds: [1, 2, 3, 4, 5]
  logging:
    server: tracking.example.internal

Common mistakes

• Using DQN for continuous actions without thought. Coarse discretization loses precision and explodes the number of actions.
• Forgetting terminal states in the target. When an episode ends, the future term must be zero.
• Sampling from a buffer that is too small or too new. Correlated data brings back the very instability replay was built to avoid.
• Updating the target network too often. The target then moves as fast as the prediction, and training wobbles.
• Unscaled inputs and rewards. Large magnitudes cause huge gradients.
• Leaking the future into the state. A feature that includes information not known at decision time makes the agent look brilliant in training and useless in practice.
• Judging by one lucky run. Results vary a lot between seeds.
• A reward that can be gamed. The agent will find the shortcut, for example by delaying jobs to avoid penalties.
• Ignoring partial observability. If the state hides history, add memory or stack observations.
• Trusting overestimated values. Rising Q-values with flat reward is a warning sign.
• Training offline without constraints. Out-of-distribution actions can get inflated values.
• No baseline. A greedy rule may already do well, and you would not know without comparing.

Frequently asked questions

What does DQN stand for and what does it do?

DQN stands for deep Q-network. It is a reinforcement-learning algorithm that uses a neural network to estimate the long-term value of each action in a given state. The agent then chooses the action with the highest estimated value, with some random exploration so that it keeps learning.

What are experience replay and the target network in DQN?

Experience replay stores past transitions in a buffer and trains on random mini-batches from it, which breaks the correlation between consecutive steps and reuses data. The target network is a delayed copy of the main network that is used to compute training targets, which keeps the targets from shifting with every update. Together they make training much more stable.

What is the Q-value?

The Q-value of a state and action is the expected total discounted reward the agent will collect if it takes that action now and then follows its policy. A higher Q-value means a better action in that state. The network learns to predict these values from experience.

Why does DQN only work with discrete actions?

The network outputs one value per action, and the agent picks the largest. With continuous actions there are infinitely many choices, so taking the maximum becomes a separate optimization problem. Methods such as DDPG, TD3 and SAC solve this by learning a policy network alongside the value function.

What is epsilon-greedy exploration?

It is a simple way to balance trying new things with using what is known. With probability epsilon the agent picks a random action, and otherwise it picks the best-valued action. Epsilon usually starts large and decays as learning progresses.

What is the difference between DQN and Double DQN?

DQN uses the same network to choose and to evaluate the best next action, which tends to overestimate values. Double DQN uses the main network to choose the action and the target network to evaluate it, which reduces the bias. The change is small in code and often improves stability.

Can DQN learn from a fixed dataset of logged decisions?

In principle yes, since it is off-policy, but it is risky. A fixed log covers only some actions, and the network may assign inflated values to actions that were never tried. Offline reinforcement-learning variants add conservative penalties or constraints to avoid this, and any such system needs careful validation before use.

How many samples does DQN need?

Usually a lot, often hundreds of thousands to millions of transitions, depending on the task. Sample efficiency improves with replay, good reward design, smaller action sets and better state features. For costly real systems, a simulator or a large log is often needed.

Is DQN suitable for tabular data problems?

It suits decision problems whose state is a row of features and whose actions are a small discrete set, such as dispatching, routing choices or intervention decisions. It is not a replacement for tabular prediction models, which are better for classification and regression on static tables. The two are often combined, with a table model supplying features or simulation.

Why do Q-values sometimes explode?

Because the update bootstraps from the network's own estimates, errors can feed back and grow. Taking a maximum over noisy estimates adds upward bias. Remedies are the target network, Double DQN, lower learning rates, gradient clipping, reward scaling and input normalization.

Glossary

• Action-value function: The function giving expected return for each state and action.
• Bellman equation: The relation that expresses the value of a state through the value of the next state.
• Bootstrapping: Using an estimate to update another estimate.
• Discount factor: The weight that reduces the value of later rewards.
• Epsilon-greedy: An exploration rule mixing random and greedy actions.
• Episode: One complete run of the agent in the environment.
• Experience replay: Storing and re-sampling past transitions.
• Huber loss: A loss that is quadratic for small errors and linear for large ones.
• Off-policy: Learning about one policy using data from another.
• Offline reinforcement learning: Learning from a fixed dataset with no further interaction.
• Overestimation bias: The tendency to inflate values when taking a maximum of noisy estimates.
• Replay buffer: The memory that holds past transitions.
• Reward: The numeric feedback signal.
• State: The agent's view of the situation.
• Target network: A delayed copy of the Q-network that supplies stable targets.
• Temporal-difference error: The gap between a prediction and its updated estimate.

Summary

DQN learns the value of each action with a neural network, trained from stored experience against a slowly changing target network, and acts by picking the best-valued action with some random exploration. Replay and the target network are the two ideas that turned an unstable combination into a workable method. It handles discrete actions well, struggles with continuous ones, and is sample hungry, but it remains a clear and strong baseline.

For work with tables, DQN is the decision layer, not the representation layer. A tabular foundation model can supply state encodings that cope with mixed types and missing values, impute gaps, flag unusual states, and help build simulators from logged data. The agent then learns what to do with those inputs. Continue with Double DQN, and see the architecture, research and benchmarks pages for the table-learning side.

Related

It belongs to Reinforcement learning in my stack, under AI and machine learning. Others in the same group: