Özer Cem KELAHMETKELAHMET
AI Simulations · Classical ML

Q-learning: an agent learns a maze from rewards alone

Reinforcement learning starts here: an agent that knows nothing about its world, a reward when it reaches the goal and a penalty when it falls into a pit. Q-learning keeps one number for every square and every move, the value of taking that move there, and improves it a little after every step it takes.

Level IntermediateAbout 10 min1 challenge in the Lab

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

Dark squares are walls, orange squares are pits (reward −1, the episode ends) and the green square is the goal (+1). Every move also costs a small step penalty. The colour of each square shows the value of the best move from it, and the arrow shows that move. The white line is the path of the last episode. The chart tracks the return per episode, smoothed over 20 episodes.

Things to try

  • Press Train with the defaults: the arrows form a path from start to goal and the success rate climbs.
  • Set exploration ε to 0 and reset: the agent may never stumble on the goal, so it never learns.
  • Lower the discount γ to 0.5: the goal's value fades within a few squares and squares far from it stay unlearned.
  • Turn on the slippery floor: the learned path moves away from the pits, because a random slip next to one costs −1.
  • Set the step cost to 0: the agent no longer hurries, and its paths get longer.

How it works in real systems

Tabular Q-learning only works when the states can be listed. DQN (2015) replaced the table with a neural network that reads the screen, and added experience replay and a target network to keep training stable. Modern systems use policy-gradient methods such as PPO as well; the same ideas of value, discount and exploration underlie the reinforcement learning used to fine-tune language models.

Key formulas

$$Q(s,a)\leftarrow Q(s,a)+\alpha\Big[r+\gamma\max_{a'}Q(s',a')-Q(s,a)\Big]$$

The Q-learning update after one move.

Common mistakes

  • Decaying exploration to zero too early: the agent locks into the first path it found.
  • A discount factor too low for the task's horizon: rewards many steps away stop counting.
  • Reward shaping that changes which policy is optimal, so the agent learns the shaped task instead of the real one.

Real-world numbers

  • Watkins introduced Q-learning in his 1989 PhD thesis; Watkins and Dayan proved its convergence in 1992.
  • DQN (2015) combined Q-learning with a deep network and reached human-level play on many Atari games.
  • Tabular Q-learning stores one value per state–action pair: this grid has 49 × 4 = 196.

Further reading

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · Tabular Q-learning on FrozenLake

Gymnasium's FrozenLake is the same kind of grid world, with a slippery floor by default.

pip install gymnasium numpy
import gymnasium as gym, numpy as np
env = gym.make('FrozenLake-v1', map_name='4x4', is_slippery=True)
Q = np.zeros((env.observation_space.n, env.action_space.n))
alpha, gamma, eps = 0.1, 0.99, 0.2
rng = np.random.default_rng(0)
wins = []
for episode in range(20_000):
    s, _ = env.reset(seed=int(rng.integers(1e9)))
    done = False
    while not done:
        a = env.action_space.sample() if rng.random() < eps else int(Q[s].argmax())
        s2, r, terminated, truncated, _ = env.step(a)
        Q[s, a] += alpha * (r + gamma * Q[s2].max() * (not terminated) - Q[s, a])
        s, done = s2, terminated or truncated
    wins.append(r)
print('success rate over the last 1,000 episodes:', np.mean(wins[-1000:]))

Step 2 · Evaluate the greedy policy

Turn exploration off and measure how often the learned policy reaches the goal.

def evaluate(n=2000):
    total = 0
    for _ in range(n):
        s, _ = env.reset()
        done = False
        while not done:
            s, r, terminated, truncated, _ = env.step(int(Q[s].argmax()))
            done = terminated or truncated
        total += r
    return total / n

print('greedy success rate:', evaluate())
print(np.array(['←↓→↑'[a] for a in Q.argmax(1)]).reshape(4, 4))

On the slippery map each move goes the intended way only one time in three, so even the best policy fails in a sizeable share of episodes. Compare the printed arrows with the map: they steer away from the holes rather than straight at the goal.

More simulations

← CLIP · All AI Simulations · Gradient boosting →

Last updated Nov 19, 2026
Type to search.
    ↑↓ navigate↵ open