Recurrent neural networks

Overview

A recurrent neural network (RNN) is a neural network built for sequences: words in a sentence, readings from a sensor, frames of audio, prices over time. Instead of seeing the whole input at once, it reads one element at a time. After each step it updates an internal vector called the hidden state, which acts as a running summary of everything it has read so far. The same weights are reused at every step, so one small network can handle sequences of any length.

For about two decades RNNs, and especially their gated forms LSTM and GRU, were the standard tool for language, speech and time series. Since 2017 Transformers have taken over most of those tasks because they train in parallel and look directly at every position. More recently, state-space models such as S4 and Mamba have revived the recurrent idea in a form that trains quickly. Understanding RNNs explains a great deal about all of these.

An everyday picture: reading a novel, you do not reread every earlier page to understand the current sentence. You carry a mental summary of the plot and characters, update it as you go, and use it to make sense of what comes next. That summary is the hidden state. If the summary is too small or fades too fast, you forget who someone was by chapter ten, which is exactly the weakness of simple RNNs.

What it is

The defining feature is the loop: the hidden state at one step is an input to the next. Everything else, gates, multiple layers, two directions, is a refinement of that loop.

RNNs can be wired for different tasks. Many-to-one reads a sequence and produces one output, as in sentiment classification. One-to-many produces a sequence from one input, as in image captioning. Many-to-many produces an output at each step, as in tagging each word, or reads one sequence and writes another, as in translation.

Where it came from

Recurrent connections appear early in neural network research. John Hopfield's 1982 network used feedback to store memories. Michael Jordan described a recurrent network for sequences in 1986, and Jeffrey Elman's 1990 paper "Finding Structure in Time" introduced the simple recurrent network that is still the textbook RNN. The training method, backpropagation through time, follows from the backpropagation work of Rumelhart, Hinton and Williams (1986) and was set out clearly by Paul Werbos in 1990.

The central difficulty was identified soon after. Sepp Hochreiter analysed vanishing gradients in his 1991 diploma thesis, and Yoshua Bengio, Patrice Simard and Paolo Frasconi published the problem in 1994. Hochreiter and Jürgen Schmidhuber answered with the long short-term memory (LSTM) network in 1997. Felix Gers, Schmidhuber and Fred Cummins added the forget gate in 2000. Mike Schuster and Kuldip Paliwal introduced bidirectional RNNs in 1997.

The 2010s were the RNN's peak. Razvan Pascanu, Tomas Mikolov and Yoshua Bengio analysed exploding gradients and popularised gradient norm clipping in 2013. Kyunghyun Cho and colleagues introduced the gated recurrent unit (GRU) and the encoder-decoder framing in 2014, and Ilya Sutskever, Oriol Vinyals and Quoc Le showed sequence-to-sequence translation with deep LSTMs the same year. Dzmitry Bahdanau, Cho and Bengio added attention to the decoder in 2014. In 2017 "Attention Is All You Need" removed recurrence entirely and the Transformer began to replace RNNs. Recurrence returned in new forms: S4 (2021), Mamba (2023), and revisits of the LSTM such as xLSTM (2024) and the minimal minLSTM and minGRU variants (2024).

How it works

The recurrent step

Plain RNN:   h_t = tanh(W_xh · x_t + W_hh · h_{t−1} + b_h)
             y_t = W_hy · h_t + b_y

LSTM:        f_t = σ(W_f · [h_{t−1}, x_t] + b_f)        forget gate
             i_t = σ(W_i · [h_{t−1}, x_t] + b_i)        input gate
             o_t = σ(W_o · [h_{t−1}, x_t] + b_o)        output gate
             c̃_t = tanh(W_c · [h_{t−1}, x_t] + b_c)     candidate memory
             c_t = f_t ⊙ c_{t−1} + i_t ⊙ c̃_t
             h_t = o_t ⊙ tanh(c_t)

Gradient through time:  ∂h_t/∂h_k = Π_{j=k+1..t} diag(tanh′(·)) · W_hh

Here x_t is the input at step t, h_t the hidden state, y_t the output, the W matrices and b vectors are learned weights and biases shared across all steps, and tanh squashes values into −1 to 1. In the LSTM, σ is the sigmoid function (output between 0 and 1, acting as a soft switch), [h_{t−1}, x_t] means the two vectors joined end to end, ⊙ is element-wise multiplication, c_t is the cell state and c̃_t is the proposed new content. The last line shows why training is hard: the gradient that links step k to a later step t is a product of t − k matrices. If those matrices tend to shrink vectors, the product vanishes; if they tend to stretch them, it explodes.

Unrolling

To train an RNN you "unroll" it: draw one copy of the network per time step, with the hidden state passed from each copy to the next. The unrolled graph is just a very deep feed-forward network in which every layer shares the same weights. That view makes training a direct application of backpropagation.

Backpropagation through time

Backpropagation through time (BPTT) runs the sequence forward, computes the loss at the outputs, then sends gradients backwards through the unrolled graph. Because weights are shared, each weight's gradient is the sum of its gradients at every step. For long sequences, full BPTT is expensive in memory, so practitioners use truncated BPTT: split the sequence into chunks of, say, 100 steps, carry the hidden state forward between chunks, but only backpropagate within each chunk. This is cheaper but cannot learn dependencies longer than the chunk.

Vanishing and exploding gradients

In a plain RNN the repeated product of Jacobians usually shrinks the signal exponentially. Gradients from step 50 back to step 1 become so small that early inputs barely affect learning. The network then only learns short-range patterns. When the product grows instead, gradients explode and a single update can throw the weights into a useless region, showing up as a sudden spike or NaN in the loss.

Gradient clipping

Clipping handles explosions. Before each update, compute the norm of the full gradient vector; if it is larger than a threshold, rescale it to that threshold. The direction is kept, only the step size is capped. A threshold around 1 to 5 is a common starting point. Clipping does nothing for vanishing gradients, which need architectural fixes.

LSTM and GRU

The LSTM adds a cell state that is updated mostly by addition rather than repeated multiplication. The forget gate decides how much of the old memory to keep, the input gate decides how much new content to write, and the output gate decides how much of the memory to expose. When the forget gate stays near 1, information and gradients can flow across many steps almost unchanged. A common trick is to initialise the forget-gate bias to a positive value so the network starts by remembering. The GRU merges the cell and hidden state and uses two gates, update and reset. It has fewer parameters and often performs similarly. Details are on the LSTM and GRU pages.

Bidirectional and stacked RNNs

A bidirectional RNN runs one RNN forwards and another backwards over the sequence and joins their states, so each position sees both past and future context. This suits tagging and classification of complete sequences but not real-time prediction, where the future is unknown. Stacked RNNs feed the hidden states of one layer as inputs to the next, building more abstract features; two to four layers are typical, often with dropout between them.

Sequence-to-sequence

An encoder RNN reads the input sequence into its final hidden state. A decoder RNN starts from that state and generates the output one token at a time, feeding back its own previous output. During training, "teacher forcing" feeds the true previous token instead. The weakness is that one fixed vector must hold the whole input. Attention solved this by letting the decoder look back at every encoder state, and that idea led directly to the Transformer.

Why Transformers and state-space models took over

An RNN must compute step t before step t + 1, so training cannot be parallelised across time on a GPU. Transformers compute all positions at once and connect any two positions in one step, which made very large language models practical. Their cost is attention that grows with the square of the sequence length and a key-value cache that grows during generation. State-space models such as S4 and Mamba keep a fixed-size recurrent state for generation, like an RNN, but are designed so that training can run in parallel using convolutions or parallel scans. In effect they recover the RNN's cheap inference without its slow training.

Where RNNs still fit

RNNs remain sensible for streaming data where each new reading must update a prediction with constant memory and latency, for small on-device models, for low-power hardware, and for many industrial time series where sequences are short and data is modest. They are also common inside larger systems, for example as a small recurrent layer in a forecasting model.

Key concepts and terms

A worked example

Suppose you want to predict the next hour's electricity load for a building from the last 48 hourly readings plus temperature and hour-of-day. This is an illustrative setup.

1. Each step's input x_t has 3 values: load, temperature and hour (encoded as sine and cosine, making 4 values in practice). Scale each to roughly zero mean and unit variance.
2. Build training windows: 48 steps in, 1 value out. Two years of hourly data gives about 17,500 windows. Split by time, not randomly: the last three months are the test set.
3. Use a two-layer LSTM with hidden size 64, dropout 0.2 between layers, and a linear output on the final hidden state. That is roughly 51,000 parameters.
4. Train with mean squared error, the Adam optimiser, learning rate 0.001, batch size 128, and gradient norm clipping at 1.0.
5. Compare with a seasonal naive baseline (the value 24 hours earlier). If the LSTM does not clearly beat it, something is wrong with the data or setup.

A PyTorch sketch of the model and one training step:

import torch
from torch import nn

class LoadLSTM(nn.Module):
    def __init__(self, n_in=4, hidden=64):
        super().__init__()
        self.lstm = nn.LSTM(n_in, hidden, num_layers=2,
                            batch_first=True, dropout=0.2)
        self.head = nn.Linear(hidden, 1)

    def forward(self, x):            # x: (batch, 48, n_in)
        out, (h_n, c_n) = self.lstm(x)
        return self.head(out[:, -1]) # use the last time step

model = LoadLSTM()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
loss = nn.functional.mse_loss(model(xb), yb)
opt.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
opt.step()

At deployment the model can run in streaming mode: keep the hidden and cell state, feed each new hourly reading, and get an updated forecast with constant memory.

Strengths and limitations

Strengths:

• Handles sequences of any length with a fixed number of parameters.
• Constant memory and compute per step at inference, ideal for streaming.
• Natural fit for ordered data and online updates.
• Gated variants learn dependencies over hundreds of steps.
• Small models train well on modest datasets.

Limitations:

• Training is sequential in time, so it uses GPUs poorly compared with Transformers.
• Very long-range dependencies remain hard even with gates.
• The fixed-size state is a bottleneck for tasks that need exact recall of earlier content.
• Sensitive to scaling, initialisation and learning rate; clipping is usually needed.
• Bidirectional variants cannot be used for real-time prediction.

How it compares with alternatives

Practical guidance and best practices

1. Start with a GRU or LSTM, not a plain RNN; plain RNNs are mainly for understanding.
2. Scale inputs and targets; recurrent networks are sensitive to large raw values.
3. Always clip gradient norms, with a threshold around 1 as a first try.
4. Split time series by time, never randomly, to avoid leaking the future into training.
5. Compare against simple baselines such as last value, seasonal naive or gradient-boosted trees on lag features.
6. Use truncated BPTT with stateful batches for very long sequences, and reset state between independent sequences.
7. Add dropout between stacked layers rather than on the recurrent connections, unless you use a variant designed for it.
8. Use bidirectional layers only when the whole sequence is available at prediction time.
9. For long contexts or large-scale language work, consider a Transformer or state-space model instead.

Common mistakes

• Shuffling time-series windows across the train and test split, which leaks future information.
• Forgetting to reset or detach the hidden state between batches, causing memory growth or cross-sequence leakage.
• Training without gradient clipping and blaming the architecture for exploding losses.
• Using a bidirectional model for live forecasting.
• Feeding unscaled inputs with very different ranges.
• Reading the output at the wrong position when sequences are padded to different lengths.

Frequently asked questions

Is an LSTM a kind of RNN?

Yes. An LSTM is a recurrent network whose step function uses gates and a separate cell state. It still reads the sequence one step at a time and carries memory forward. The same is true of the GRU.

Why do gradients vanish in RNNs?

The gradient from a late step to an early one is a product of many factors, one per step in between. If those factors are usually smaller than one in size, the product shrinks exponentially. The tanh activation, whose slope is at most one, and recurrent weights with small singular values both push in that direction.

Does gradient clipping fix vanishing gradients?

No. Clipping only caps gradients that are too large. Vanishing gradients need changes to the architecture, such as LSTM or GRU gates, careful initialisation of the recurrent weights, or skip connections through time.

Are RNNs obsolete?

Not entirely. For large language models and most long-sequence tasks, Transformers and state-space models are preferred. RNNs remain practical for streaming data, small datasets, low-latency edge devices and as components inside larger models. Recent research has also revisited simplified and extended LSTMs that train more efficiently.

How is a state-space model different from an RNN?

Both carry a fixed-size state forward. Classic RNNs use a non-linear update that must run step by step. State-space models use structured updates that can be computed in parallel during training, then run step by step for fast generation. Mamba adds input-dependent ("selective") updates so the model can choose what to keep.

What is teacher forcing?

During training of a sequence generator, the decoder is fed the true previous token rather than its own prediction. This makes training faster and more stable. The downside is a gap between training and use, since at inference the model must consume its own possibly wrong outputs.

Glossary

• RNN: a network that applies the same step function across a sequence, carrying a hidden state.
• Hidden state: the memory vector updated at each step.
• BPTT: backpropagation through time over the unrolled network.
• LSTM: a gated RNN with a separate cell state and forget, input and output gates.
• GRU: a gated RNN with update and reset gates and no separate cell state.
• Forget gate: the LSTM gate that decides how much old memory to keep.
• Bidirectional RNN: two RNNs reading forwards and backwards, combined.
• Encoder-decoder: an architecture that reads one sequence and writes another.
• Gradient clipping: capping the gradient norm to prevent unstable updates.
• State-space model: a recurrent-style model designed for parallel training.

Summary

A recurrent neural network reads a sequence step by step and keeps a hidden state that summarises the past, reusing the same weights at every step. It is trained by backpropagation through time on the unrolled network, which exposes the classic problems of vanishing and exploding gradients. Gradient clipping tames explosions, and gated cells such as the LSTM and GRU let information survive across long spans. Bidirectional, stacked and encoder-decoder variants extended RNNs to tagging, translation and speech. Transformers replaced them for most large-scale work because they train in parallel and attend to every position, and state-space models now offer recurrent-style inference with parallel training. RNNs still fit streaming, small-data and on-device problems, and their core ideas, a compact memory and a learned update, run through much of modern sequence modelling. For the ideas they build on, see backpropagation and gradient descent.

Related

It belongs to Architectures in my stack, under Deep learning. Others in the same group: