GRU

Overview

A GRU, or gated recurrent unit, is a recurrent neural network cell that reads a sequence step by step and keeps a single hidden state as its memory. It uses two gates, an update gate and a reset gate, to decide how much of the past to keep and how much of the new input to mix in. It is a simpler cousin of the LSTM, with fewer parameters and often a faster training time.

In many tasks a GRU reaches accuracy close to an LSTM. It is popular for time series, sensor data, small language tasks and any setting where compute or memory is tight. It is also a useful building block for ordered rows in a table, where each entity has a history of events. This page explains what a GRU is, how it works, how to use it, and how its ideas connect to models that learn from raw tables.

What it is

A GRU is a recurrent layer. "Recurrent" means that the same small network is applied at every time step, and its output at one step is fed back as input at the next. This lets it handle sequences of any length with a fixed set of weights.

The GRU keeps one vector of memory, called the hidden state. At every step it asks two questions. How much of the old memory should I keep? How much of the old memory should I look at when building a new candidate? The answers come from two gates.

The two gates

• The update gate decides how much of the previous hidden state is carried forward and how much is replaced by new content.
• The reset gate decides how much of the previous hidden state is used when computing the new candidate content.

Both gates output numbers between 0 and 1 for each memory slot. A value near 1 means "yes, use it" and a value near 0 means "no, ignore it".

A note-taking analogy

Imagine a student taking notes in a lecture, using a single page. The update gate is the choice between keeping the current page as it is and rewriting it with fresh material. The reset gate is the choice of whether to glance at the old page before writing a new line, or to start from a blank mind. A student who sets the reset gate low behaves as if the topic just changed. A student who sets the update gate high leaves the page almost untouched while a short digression passes by.

How it differs from an LSTM

The coupling is the key design choice. In a GRU, the amount kept and the amount written add up to one. In an LSTM they are controlled separately.

Where it came from

Recurrent networks learn by backpropagation through time, and plain versions struggle with long dependencies because gradients vanish or explode. The LSTM, introduced in 1997, addressed this with a gated cell state. It worked, but it was heavy.

The GRU was introduced in 2014 in work on neural machine translation, in the same period that sequence-to-sequence models were taking shape. The authors wanted a cell that adapts to different time scales without the extra machinery of the LSTM. Soon after, empirical studies compared the two on music modelling and speech tasks. They found that gated units clearly beat plain recurrent units, and that GRU and LSTM were close to each other, with the winner depending on the task.

Through the mid-2010s GRUs were used in speech, translation, captioning and time-series work. After 2017, the transformer described in Transformers took over many language and vision problems. Recurrent units stayed in use for streaming, edge devices and small data. They also influenced newer designs such as the state-space models S4 and Mamba, which use input-dependent gating to select what to remember. For how these lines connect to table-focused work, see /model/research.

How it works

This section follows one GRU step and then a full sequence.

Step 1: receive the inputs

At time step t the cell receives the input vector x_t and the previous hidden state h_(t-1). In a sensor table, x_t could be the readings at one moment. In an event log, it could be an embedding of the event type.

Step 2: compute the update gate

The update gate z_t is a sigmoid layer applied to the input and the previous hidden state. It produces a vector of values between 0 and 1. It answers: for each memory slot, how much should the state change at this step?

Step 3: compute the reset gate

The reset gate r_t is another sigmoid layer over the same inputs. It answers: for each memory slot, how much of the past should be visible when forming the new candidate?

Step 4: build the candidate

The candidate state is a tanh layer. Its input is x_t and the previous hidden state scaled by the reset gate. If the reset gate is near zero, the candidate depends almost only on the new input, as if the memory were wiped.

Step 5: blend old and new

The new hidden state is a mix of the old state and the candidate. The update gate sets the mix. Where z_t is near 0 the old value stays. Where z_t is near 1 the candidate takes over. Some write the mix the other way round, with the roles of z and one minus z swapped. Libraries differ, but the behaviour is equivalent.

The equations

z_t = sigma(W_z [h_(t-1), x_t] + b_z)
r_t = sigma(W_r [h_(t-1), x_t] + b_r)
n_t = tanh (W_n [r_t * h_(t-1), x_t] + b_n)
h_t = (1 - z_t) * h_(t-1) + z_t * n_t

Here sigma is the sigmoid and the star is elementwise multiplication.

Why the gradient survives

Look at the last line. The new state is a weighted sum of the old state and a candidate. Going backwards, the gradient flowing to the old state includes a term equal to one minus the update gate. When that term is near one, the gradient passes through almost unchanged, much like the cell state path in an LSTM. The network can learn to keep that path open for slots that should remember for a long time, and to close it for slots that should react quickly.

A tiny numeric walk-through

Take one memory slot. The old hidden value is 0.6. The update gate is 0.25, the reset gate is 0.5, and the candidate (already computed with the reset applied) is -0.2.

• Kept part: (1 minus 0.25) times 0.6 gives 0.45.
• Written part: 0.25 times -0.2 gives -0.05.
• New hidden value: 0.45 plus -0.05 gives 0.40.

The slot moved from 0.6 to 0.4. It changed a little, because the update gate was low. If the update gate had been 0.9, the new value would be 0.1 times 0.6 plus 0.9 times -0.2, which is 0.06 minus 0.18, or -0.12. The slot would nearly flip sign. This shows how one number controls the speed of forgetting.

Training

Training unrolls the cell over the sequence and uses backpropagation through time. Good habits include:

• Clip gradients to a maximum norm.
• Use the Adam optimiser with a modest learning rate, as outlined in Optimization.
• Apply dropout between stacked layers.
• Use truncated windows for long streams.
• Use masks or packed sequences for padded batches.

Output heads

As with other recurrent layers, you can pick what to read out.

• The last hidden state for whole-sequence classification or regression.
• The hidden state at every step for per-step labels.
• A decoder GRU for sequence-to-sequence tasks.
• A pooled state, such as mean or max over time, for noisy sequences.

Bidirectional and stacked GRUs

A bidirectional GRU runs two GRUs, one forward and one backward, and joins their states. It is helpful when the whole sequence is available, for example when imputing a gap or labelling a stored log. A stacked GRU feeds the output sequence of one layer into the next layer. Two layers are a common starting point.

Variants and extensions

Several changes to the basic GRU appear in papers and libraries.

• Minimal gated units use a single gate, which cuts parameters further.
• Light GRUs remove the reset gate and use a ReLU candidate for speed.
• GRU with decay adds a time-based fade on the state for irregular sampling.
• Convolutional GRUs use convolutions inside the gates for grid-shaped inputs such as maps or frames.
• Layer-normalised GRUs normalise the gate inputs to stabilise deeper stacks.
• Attention on top of a GRU lets the output look back at every hidden state, not only the last one.

Each variant trades a little generality for speed or stability. The plain GRU is the safe default.

Reading the gates

One of the nicest features of a GRU is that its gates can be inspected. After training, run a few sequences and record the update gate values per step.

• Slots with update values near zero for long stretches act as long-term memory.
• Slots with update values near one react quickly to each input.
• A sudden spike across many slots often marks a regime change in the data.

This is not a full explanation of the model, but it gives a window into what the cell treats as important. It can help spot data problems, such as a sensor that resets the gates every time it drops out.

Everyday picture of a full sequence

Picture a river flowing past a series of gates in a canal. The water is the hidden state. At each gate station, the keeper opens or closes the sluices based on what is arriving from upstream. Some water passes through unchanged. Some is diverted and replaced with fresh water from a side stream. The keeper never sees the whole river, only the local flow and the current level. After many stations, the water that reaches the sea carries a blend of what was kept and what was added along the way.

A project workflow in seven steps

1. Define the prediction target and the time horizon in plain words.
2. Gather ordered data per entity and check gaps, duplicates and units.
3. Build a naive baseline and a tree baseline with lag features.
4. Build the windows and the split by entity and time.
5. Train a small GRU and compare it to the baselines.
6. Tune window length, hidden size and dropout one at a time.
7. Test once on the held-out period, then freeze the pipeline and monitor drift.

Following this order prevents the common trap of tuning a complex model before knowing whether the problem is learnable at all.

Key concepts and terms

A worked example

This example forecasts the next hour of a utility load from the previous day. The data is synthetic and all names are placeholders.

The task

Each sample has 24 hourly steps. Each step has 4 features: load, temperature, hour of day as sine, and hour of day as cosine. The target is the load at the next step.

Model

import torch
import torch.nn as nn

class LoadGRU(nn.Module):
    def __init__(self, n_features=4, hidden=48, layers=2, dropout=0.1):
        super().__init__()
        self.gru = nn.GRU(
            input_size=n_features,
            hidden_size=hidden,
            num_layers=layers,
            batch_first=True,
            dropout=dropout,
        )
        self.head = nn.Linear(hidden, 1)

    def forward(self, x):
        out, h_n = self.gru(x)        # a GRU returns only a hidden state
        return self.head(out[:, -1]).squeeze(-1)

model = LoadGRU()
x = torch.randn(16, 24, 4)
print(model(x).shape)                 # torch.Size([16])

Note that a GRU returns only a hidden state, while an LSTM also returns a cell state. This small difference simplifies state handling when you serve predictions.

Training

opt = torch.optim.Adam(model.parameters(), lr=2e-3)
sched = torch.optim.lr_scheduler.ReduceLROnPlateau(opt, patience=3)
loss_fn = nn.L1Loss()

for epoch in range(30):
    model.train()
    for xb, yb in train_loader:
        opt.zero_grad()
        loss = loss_fn(model(xb), yb)
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
    model.eval()
    with torch.no_grad():
        val = sum(loss_fn(model(x), y).item() for x, y in val_loader) / len(val_loader)
    sched.step(val)

Data steps

1. Sort the series by time and check for gaps.
2. Create cyclic features for hour and weekday.
3. Scale numeric columns with training statistics only.
4. Build windows of 24 steps with the next value as target.
5. Split by time: the oldest 70 percent for training, the next 15 percent for validation, the last 15 percent for testing.

Baselines to beat

Before trusting the model, compare it with simple baselines.

• Repeat the previous hour.
• Repeat the same hour of yesterday.
• Repeat the same hour of last week.
• A tree model with lag features, for example XGBoost.

If the GRU does not beat the same-hour-yesterday baseline on held-out weeks, the model is not yet earning its keep.

A second example: parameter and speed comparison

Take a hidden size of 64 and an input size of 8. One GRU layer has roughly three blocks of weights, each of size hidden times (input plus hidden) plus a bias. That is about 3 times (64 times 72 plus 64), which is close to 14,000 weights. An LSTM layer of the same size has four blocks, about 18,500 weights. The GRU is about a quarter smaller. On a CPU, with the same batch and window, the GRU usually runs a bit faster. Exact timing depends on the library and hardware, so measure on your own setup.

What good looks like

A healthy training run has a few visible signs.

• Training loss falls quickly in the first epochs, then slowly.
• Validation loss follows training loss with a small gap.
• Predictions on held-out days track the shape of the true curve.
• Errors are spread evenly over hours, with no single hour far worse.
• Results vary only a little across seeds.

A large gap between training and validation loss points to overfitting. Fixes include more dropout, a smaller hidden size, more data, or fewer input features.

Reading the result

Plot predictions against truth for a few held-out days. Check the error by hour. Models often do well in stable periods and miss sudden peaks. If so, add weather forecasts or holiday flags as inputs, rather than making the network larger.

Strengths and limitations

Strengths

• Fewer parameters than an LSTM, so lower memory use and faster steps.
• Often similar accuracy to an LSTM across many tasks.
• A single state vector makes serving and caching simple.
• Works well on small and medium datasets.
• Good fit for streaming inference, because only the last state is needed.
• Easy to implement and supported in every major framework, including PyTorch.

Limitations

• Sequential computation limits parallel training compared with attention models.
• Very long dependencies still fade, so extremely long contexts need other designs.
• Less fine control than an LSTM, because keep and write are coupled.
• No direct access to distant steps, since all information passes through one state.
• Needs ordered, regularly spaced input, or explicit time-gap features.
• Not designed for unordered columns in a single table.
• Interpretation is limited, although attribution tools like SHAP can help.

How it compares with alternatives

GRU or LSTM

There is no universal winner. A GRU is a good first choice when you want speed and a small model. An LSTM is worth trying when the task needs more careful memory control or when the sequence is long. Many teams train both and keep the better one on validation data. When the two differ by a hair, pick the GRU for its lower cost.

GRU or transformer

A transformer needs more data to shine and uses memory that grows with the square of the sequence length in its basic form. A GRU uses constant memory per step. For short windows and limited data, a GRU is often competitive and far cheaper.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types in its columns, many missing cells, and sometimes no useful column names. See /model/architecture for how such a model is built. A GRU relates to this world in several concrete ways.

Rows as ordered events

Many tables are logs: each row is a time-stamped event for some entity. A GRU can read each entity's rows in order and produce a summary vector. That vector can be added as a feature for a table model such as TabPFN or FT-Transformer. In this way the GRU handles order, and the table model handles the rest of the columns.

Mixed types

A GRU takes numeric vectors. Categorical columns therefore become learned embeddings, and numbers are scaled. A tabular foundation model builds this into its encoder, so it does not need hand-made preprocessing. It encodes every cell by type, and the GRU analogy is a way to see how one row at a time becomes a vector.

Missing values with decay

A well-known GRU variant for clinical and sensor tables adds a decay on the hidden state based on the time since the last observation, together with a mask for which values were seen. The idea is to trust the old state less as time passes. Tabular foundation models share the spirit: they carry a missing marker for each cell and learn how to treat it. See Imputation Networks and Masked Cell Modeling.

No column names

A GRU sees positions, not names. If features sit in an arbitrary order, a recurrent pass over features would depend on that order. This is one reason table models use attention across columns that does not depend on order, as in SAINT and Axial Attention. Recurrent layers are best kept for the time axis, not the column axis.

Pre-training and in-context learning

A GRU is usually trained for one task on one dataset. A tabular foundation model is pre-trained on many tables, often generated synthetically, and then predicts for a new table by reading labelled rows as context, without weight updates. A GRU's hidden state is a small, fixed-size memory that adapts as it reads a sequence. Attention-based table models turn that idea into a large explicit memory over rows. See /model/research for the research background.

Continual learning

When a recurrent model is fine-tuned on new data, it can forget old patterns. The same risk applies to table models that keep learning from fresh customers or machines. Methods such as EWC and Experience Replay reduce forgetting, and they apply to GRU encoders in the same way.

Imputation

A bidirectional GRU can fill gaps in a time series by reading context on both sides. Tabular foundation models do something related at larger scale: random cells are hidden during pre-training, and the model learns to predict them. The result is an imputer that works across many tables.

Anomaly detection

Train a GRU to predict the next step, or to rebuild a window as an autoencoder. A large error flags unusual behaviour. This links to Anomaly Detection and Autoencoders. In a tabular foundation model, a similar signal can come from how surprising a row is given the rest of the table.

Sector and semantic classification

When a table must be classified by meaning, such as what kind of process it records, the text of headers and values carries the signal. Older systems encoded strings with a character-level GRU. Modern table encoders use attention for this job, but the pattern of turning cells into vectors is the same.

Practical guidance and best practices

Data

• Split by entity and by time, never by random window.
• Fit scalers on training data only.
• Add cyclic features for hour, day and month.
• Add a mask and a time-gap feature when sampling is irregular.
• Keep windows long enough to cover at least one full cycle of the pattern you care about.

Model

• Start with one or two layers and 32 to 128 hidden units.
• Add dropout between layers.
• Use bidirectional layers only when the full window is known at prediction time.
• Compare against an LSTM and a simple tree baseline.
• Keep a dense head small.

Training

• Clip gradients from the start.
• Reduce the learning rate on a plateau.
• Stop early on a validation split of unseen entities.
• Run several seeds and report the spread.
• Use mixed precision if the sequence or batch is large, as in Mixed Precision.

Hyperparameters

A search tool such as Optuna can explore these ranges. Keep a separate test period that is never used in tuning.

Debugging checklist

When a GRU does not learn, check these in order.

• Does the model overfit a tiny batch of 16 samples? If not, there is a bug in shapes or labels.
• Are inputs scaled to a similar range? Unscaled features can saturate the gates.
• Is the target scaled? A target in the thousands makes the loss huge at the start.
• Are windows aligned with labels, with no off-by-one error?
• Is the loss on padded steps masked?
• Is the learning rate too high? Try a tenth of it.
• Is the window long enough to hold the signal?
• Does a trivial baseline already match the model? If so, the signal may be weak.

Choosing the window and horizon

The window is how far back the model looks. The horizon is how far ahead it predicts. A good rule is to set the window to two or three times the longest cycle that matters. For hourly load with daily and weekly cycles, a window of 48 to 168 steps is typical. Longer horizons are harder, so consider predicting several steps at once with a multi-output head, or predicting step by step and feeding each output back in. The second method accumulates error, so check it on held-out data.

Deployment

• Store one hidden vector per entity for streaming use.
• Reset it when a session ends.
• Export to a portable runtime such as ONNX Runtime when you need fast inference.
• Monitor input drift and retrain on a schedule.

Sizing a GRU for a budget

Start at the smallest size that beats your baseline and grow only if validation error keeps falling.

Common mistakes

• Random splitting of windows, which leaks near-duplicate data into the test set.
• Fitting scalers on all data, which leaks future statistics.
• Forgetting masks for padded batches, so padding is learned as signal.
• Taking the last padded step instead of the last real step.
• Using a bidirectional layer for real-time prediction, where the future is unknown.
• Carrying hidden state across shuffled batches.
• Skipping naive baselines and over-claiming gains.
• Making the network bigger when better features would help more.
• Ignoring exploding gradients, which appear as NaN losses.
• Leaving dropout on during evaluation.
• Using a GRU across unordered columns of a plain table.

Frequently asked questions

What does GRU stand for?

GRU stands for gated recurrent unit. It is a recurrent neural network cell that uses gates to control how a hidden state is updated as a sequence is read, so that useful information can be kept across many steps.

How many gates does a GRU have?

A GRU has two gates: an update gate and a reset gate. The update gate sets how much of the state is replaced at each step. The reset gate sets how much of the past is used when forming the new candidate content.

Is GRU better than LSTM?

Neither is always better. A GRU is smaller and usually faster, and it matches an LSTM on many tasks. An LSTM can do better on tasks that need fine memory control or long spans. The practical answer is to test both on your validation data, as discussed on the LSTM page.

Why does a GRU train faster?

It has fewer weight blocks per layer and no separate cell state. That means fewer operations at each step and less memory to move. The speed gain is modest, and on some hardware the two are close, so it is worth measuring.

Can a GRU handle very long sequences?

It handles medium spans well, but information still fades over very long ranges. For thousands of steps, attention or state-space models such as Mamba are usually a better choice, or you can summarise the data into coarser steps first.

Can I use a GRU for tabular data?

Yes, when rows form an ordered sequence per entity, such as events over time. For a single unordered table of independent rows, tree models or a tabular foundation model like TabPFN are usually a better fit.

How do I handle missing values with a GRU?

Add a mask that marks which entries were observed, fill the gaps with a simple value, and provide the time since the last observation. Some variants decay the hidden state as the gap grows. These steps let the model learn how much to trust stale information.

Is a GRU still relevant after transformers?

Yes. It remains useful for small data, streaming inference, embedded devices and tasks with short to medium memory. It also teaches the gating ideas that appear in newer sequence models.

What is the difference between the update gate and the reset gate?

The update gate decides how much of the old state to keep versus replace. The reset gate decides how much of the old state to look at when making the new candidate. One controls the output blend, and the other controls the input to the candidate.

How can a GRU help with anomaly detection?

Train it to predict the next reading or to rebuild a window of normal data. When new data produces a large error, it does not match learned behaviour and can be flagged. Set the threshold using validation data and review alerts with domain knowledge.

Glossary

• Backpropagation through time: gradient training that unrolls a recurrent network across its steps.
• Bidirectional: reading a sequence forward and backward.
• Candidate state: the proposed new memory content built at each step.
• Dropout: randomly zeroing units during training to reduce overfitting.
• Exploding gradient: gradients that grow without bound.
• Gate: a sigmoid layer that outputs values between 0 and 1 to control information flow.
• Gradient clipping: capping gradient size to stabilise training.
• Hidden state: the memory vector carried from step to step.
• Mask: a flag that marks padded or missing entries.
• Recurrent neural network: a network that reuses the same weights at each time step.
• Reset gate: the gate that controls how much past is used for the candidate.
• Sigmoid: a function that maps values into the range 0 to 1.
• Tanh: a function that maps values into the range -1 to 1.
• Truncated BPTT: backpropagation limited to fixed-length windows.
• Update gate: the gate that controls how much the state changes.
• Vanishing gradient: gradients that fade to nearly zero over many steps.

Summary

A GRU is a compact gated recurrent unit with an update gate and a reset gate. It keeps one hidden state, blends old and new content with a learned mix, and passes gradients through time much like an LSTM does, but with fewer parameters. It trains quickly, works on small and medium datasets, and suits streaming use.

For tabular work, a GRU shines when rows form ordered histories. It can summarise those histories for a downstream table model, fill gaps with a bidirectional pass, and flag anomalies through prediction error. Tabular foundation models extend related ideas by pre-training on many tables and learning in context, as described in /model/architecture, /model/research and /model/benchmarks. Start with a GRU as a fast baseline, compare it with an LSTM, and always split by entity and by time.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: