LSTM

Overview

LSTM stands for long short-term memory. It is a type of recurrent neural network that reads a sequence one step at a time and keeps a memory that survives for many steps. Gates inside the cell decide what to write into memory, what to erase, and what to show to the next layer. This design fixes a training problem that stopped plain recurrent networks from learning long-range patterns.

LSTM networks were the standard tool for sequences for almost two decades. They power sensor forecasting, speech models, text models and remaining-useful-life estimation. Transformers such as the ones described in Transformers have taken over many language tasks, yet LSTM still wins in small-data, streaming and low-latency settings. For tabular work, an LSTM matters when rows arrive as ordered events. The ideas behind its gates also echo in modern state-space models such as Mamba and S4.

What it is

An LSTM is a neural network layer built around a repeating cell. At each time step the cell receives the current input and two vectors from the previous step. One vector is the hidden state, which is the cell's short-term output. The other is the cell state, which is the long-term memory.

The cell state runs through the whole sequence like a conveyor belt. Small, controlled edits are made to it at each step. Because the edits are additive and gated, information can ride along the belt for hundreds of steps without being crushed.

The three gates

An LSTM cell has three gates and one candidate update. Each gate is a small neural layer with a sigmoid output between 0 and 1. A value near 0 means "block" and a value near 1 means "let through".

• The forget gate decides how much of the old cell state to keep.
• The input gate decides how much of the new candidate content to write.
• The output gate decides how much of the cell state to expose as the hidden state.
• The candidate update proposes new content using a tanh layer.

A kitchen analogy

Think of a cook who keeps a notebook while preparing a long meal. The forget gate is the habit of crossing out notes that are no longer useful, such as "oven is preheating" once it is hot. The input gate is the choice of which new facts deserve a line in the notebook. The output gate is the choice of which notes to read aloud to the helper right now. The notebook is the cell state. The spoken words are the hidden state.

Variants

Several variants exist, and you will see them in libraries.

• Peephole connections let the gates look at the cell state directly.
• Bidirectional LSTMs run one pass forward and one backward and join the results.
• Stacked LSTMs place several layers on top of each other.
• Projected LSTMs shrink the hidden state to save parameters.
• Convolutional LSTMs replace matrix products with convolutions for grid-shaped inputs.

The GRU is a lighter relative that merges some gates and drops the separate cell state.

Where it came from

Recurrent networks were studied through the 1980s. They learned by backpropagation through time, which unrolls the network over the sequence and passes error signals backwards. Researchers found that these error signals tended to shrink or blow up as they travelled back through many steps. This is called the vanishing and exploding gradient problem. A network that cannot pass error signals back far cannot learn that an event at step 5 explains an outcome at step 500.

The LSTM was introduced in 1997 to solve exactly this. The core idea was a "constant error carousel": a path through time where the gradient is multiplied by roughly one at each step, so it neither vanishes nor explodes. Gates were added to control access to that path.

The forget gate was added a few years later. The original design could only accumulate memory, which made long streams hard to reset. Letting the cell erase its own state made it practical for continuous input.

During the 2010s LSTMs became the default for speech recognition, handwriting recognition and machine translation. Sequence-to-sequence models, which pair an encoder LSTM with a decoder LSTM, set the standard for translation around 2014. Attention was first added on top of recurrent encoders and then, in the 2017 paper titled "Attention Is All You Need", used on its own. That paper introduced the transformer and shifted the field. See /model/research for how this lineage connects to current table-focused work.

How it works

This section walks through one step of an LSTM cell and then through a full sequence.

Step 1: receive the inputs

At time step t the cell sees the input vector x_t. It also sees the previous hidden state h_(t-1) and the previous cell state c_(t-1). The input is often a row of numbers, such as a few sensor readings at one moment.

Step 2: decide what to forget

The forget gate looks at x_t and h_(t-1). It outputs a vector of numbers between 0 and 1, one per memory slot. The old cell state is multiplied by this vector. Slots with a value near 0 are wiped. Slots near 1 are kept.

Everyday analogy: a shop assistant glances at the day's news and the current till total, then decides which old sticky notes can go in the bin.

Step 3: decide what to write

The input gate also looks at x_t and h_(t-1) and outputs numbers between 0 and 1. In parallel, a tanh layer builds a candidate vector of new content with values between -1 and 1. The two are multiplied. The result is the amount of new content that is allowed in.

Step 4: update the cell state

The new cell state is the forgotten old state plus the gated new content. This is a sum, not a repeated multiplication by a weight matrix. That sum is why gradients survive the trip backwards through time.

Step 5: produce the output

The output gate looks at x_t and h_(t-1). A tanh is applied to the new cell state, and the result is multiplied by the output gate. This gives the new hidden state h_t, which is sent to the next layer and to the next time step.

The equations

For readers who want the compact form, here is the standard cell. The symbol sigma is the sigmoid and the dot marks elementwise multiplication.

f_t = sigma(W_f [h_(t-1), x_t] + b_f)
i_t = sigma(W_i [h_(t-1), x_t] + b_i)
g_t = tanh (W_g [h_(t-1), x_t] + b_g)
o_t = sigma(W_o [h_(t-1), x_t] + b_o)
c_t = f_t * c_(t-1) + i_t * g_t
h_t = o_t * tanh(c_t)

Training with backpropagation through time

Training unrolls the cell over the sequence. A loss is computed on the outputs, and gradients flow backwards through every step. For long sequences, training uses truncated backpropagation through time: the sequence is cut into windows of fixed length, and gradients are propagated only inside each window. The hidden and cell states may still be carried forward between windows so memory is not lost.

Common practice includes:

• Gradient clipping, which caps the gradient norm to stop explosions.
• Setting the forget gate bias to a positive value at the start so the cell begins by remembering.
• Dropout applied to the vertical connections between layers.
• Optimisers such as Adam, covered in general terms in Optimization.
• Mixed precision to speed up training, as in Mixed Precision.

Handling variable length

Real sequences differ in length. Libraries pad short sequences to a common length and pass a mask, or they use packed sequences that skip the padding. Without masking, the network learns from padding as if it were data, which hurts accuracy.

Output heads

The same cell can feed different heads.

• Many-to-one: use the final hidden state to classify the whole sequence or to predict one value.
• Many-to-many aligned: predict a label at each step, such as an anomaly flag per reading.
• Sequence-to-sequence: an encoder reads the input, a decoder writes a different sequence.
• Forecasting: predict the next k values from the last n values.

Why the gradient survives

In a plain recurrent network the hidden state at each step is a squashed function of the previous one. Going backwards, the gradient is multiplied by the same weight matrix and by a squashing slope again and again. If the typical factor is below one, the signal dies. If it is above one, it explodes.

In an LSTM the cell state follows the rule new = forget times old plus gated new content. Going backwards through that rule, the gradient with respect to the old cell state is the forget gate value. When the forget gate is close to one, the gradient passes almost unchanged. The network can therefore keep a learning path open for as long as the forget gate stays high.

A simple picture: a plain RNN is like passing a message through a chain of people who each retell it in their own words. An LSTM is like handing a sealed envelope along the chain, with each person allowed to add or remove a note under strict rules.

Counting parameters

An LSTM layer has four sets of weights, one for each gate and one for the candidate. For an input of size n and a hidden size m, one layer has about four times (m times (n plus m) plus m) parameters.

• Input size 6 and hidden size 64: about 4 times (64 times 70 plus 64), which is roughly 18,000 weights.
• Input size 64 and hidden size 64 for a second layer: about 33,000 weights.
• Doubling the hidden size roughly quadruples the recurrent part.

Knowing this helps when you budget memory on an edge device.

Shapes in a framework

Most libraries use three-dimensional tensors. The common layout is batch, time, features. A few things trip people up.

• Some libraries default to time first. Check the batch-first flag.
• The returned hidden and cell states have a layer dimension, and a bidirectional layer doubles it.
• The output sequence contains the hidden state at every step, while the final state tuple holds only the last step of each layer.
• For a classification head, take the last valid step, not the last padded step.

Stateful versus stateless use

A stateless setup resets the memory at the start of every window. It is simple and safe, because each window is independent. A stateful setup carries memory from one window to the next in order. It is useful for one very long stream, but it needs sequential batches and careful resets at boundaries. Most projects should start stateless.

Key concepts and terms

A worked example

This example builds a small LSTM that predicts the remaining useful life of a machine from a window of sensor readings. All data below is synthetic and the names are placeholders.

Setup

Each sample is a window of 30 time steps. Each step has 6 sensor values, such as temperature, vibration and pressure. The target is the number of cycles left before failure, scaled to the range 0 to 1.

import torch
import torch.nn as nn

class RulLSTM(nn.Module):
    def __init__(self, n_features=6, hidden=64, layers=2, dropout=0.2):
        super().__init__()
        self.lstm = nn.LSTM(
            input_size=n_features,
            hidden_size=hidden,
            num_layers=layers,
            batch_first=True,
            dropout=dropout,
        )
        self.head = nn.Sequential(
            nn.Linear(hidden, 32),
            nn.ReLU(),
            nn.Linear(32, 1),
        )

    def forward(self, x):
        # x: (batch, time, features)
        out, (h_n, c_n) = self.lstm(x)
        last = out[:, -1, :]          # hidden state at the last step
        return self.head(last).squeeze(-1)

model = RulLSTM()
x = torch.randn(8, 30, 6)             # fake batch
print(model(x).shape)                 # torch.Size([8])

Training loop

opt = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()

for epoch in range(20):
    for xb, yb in loader:             # loader yields (batch, 30, 6) and (batch,)
        opt.zero_grad()
        pred = model(xb)
        loss = loss_fn(pred, yb)
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()

Preparing the data

The quality of the windows matters more than the size of the network. Follow these steps.

1. Sort each machine's readings by time.
2. Scale each sensor using statistics from the training machines only.
3. Slide a window of 30 steps across each machine's history with a stride of 1 or more.
4. Label each window with the remaining cycles at its last step.
5. Split by machine, not by window, so no machine appears in both train and test.

A tiny numeric walk-through

To see the gates act, take one memory slot with a single input. Suppose the old cell value is 0.8. The forget gate outputs 0.9, the input gate outputs 0.5 and the candidate is -0.4.

• Kept memory: 0.9 times 0.8 gives 0.72.
• New content: 0.5 times -0.4 gives -0.20.
• New cell state: 0.72 plus -0.20 gives 0.52.

If the output gate is 0.7, the hidden state is 0.7 times tanh(0.52). Since tanh(0.52) is about 0.48, the hidden state is about 0.34. The memory moved only a little, because the gates were gentle. If the forget gate had been 0.1, the old value would almost vanish. That is how the cell learns to reset when a new regime starts.

A second example: classifying event sequences

The same cell works for classification. Suppose each sample is a session of up to 50 events, and each event is a category such as "login", "view" or "pay". The task is to flag sessions that look fraudulent.

class SessionClassifier(nn.Module):
    def __init__(self, n_events=40, emb=16, hidden=48):
        super().__init__()
        self.emb = nn.Embedding(n_events, emb, padding_idx=0)
        self.lstm = nn.LSTM(emb, hidden, batch_first=True, bidirectional=True)
        self.out = nn.Linear(hidden * 2, 1)

    def forward(self, ids, lengths):
        x = self.emb(ids)
        packed = nn.utils.rnn.pack_padded_sequence(
            x, lengths.cpu(), batch_first=True, enforce_sorted=False
        )
        _, (h_n, _) = self.lstm(packed)
        h = torch.cat([h_n[-2], h_n[-1]], dim=-1)  # forward and backward finals
        return self.out(h).squeeze(-1)

Points to notice:

• The embedding layer turns a category into a learned vector, the same trick tabular models use for categorical columns.
• Packing skips padding, so the final state belongs to the last real event.
• The two final states from the two directions are joined before the output layer.
• Class imbalance is likely. Use a weighted loss or sample the rare class more often.

Reading the result

Plot predicted versus true remaining life for held-out machines. A good model tracks the downward slope and becomes more accurate near failure, because late-life sensor drift is easier to see. Report an error such as root mean squared error, and also an asymmetric score if late predictions are costlier than early ones.

Strengths and limitations

Strengths

• It handles sequences of any length with a fixed number of parameters.
• Gated memory lets it learn dependencies across hundreds of steps.
• It processes data in a stream, so inference needs only the last state and the new input. This suits edge devices.
• It works well with small and medium datasets where a transformer would overfit.
• It is well supported in every major framework, including PyTorch.
• It is robust to irregular noise, since gates can suppress useless inputs.

Limitations

• Training is sequential across time, so it parallelises poorly compared with attention.
• Very long dependencies, such as thousands of steps, still fade.
• It has no direct access to far-away steps. Everything must pass through the memory vector.
• Each cell has about four times the weights of a plain recurrent cell, which raises cost.
• It needs regular time steps or explicit time features. Irregular sampling needs extra handling.
• Interpretation is harder than with a linear model, though tools like SHAP can help with attribution.
• It does not natively understand tables with unordered columns. It expects ordered steps.

How it compares with alternatives

Choosing between them

Ask three questions. How long are the dependencies? How much data do you have? Do you need streaming inference? For a short series with a clean trend, ARIMA or lag features with XGBoost are strong baselines. For many sensor streams with mid-range memory and limited data, LSTM is a sound pick. For very large corpora or very long context, attention or state-space models usually win.

A quick decision guide

• Choose LSTM when memory of tens to hundreds of steps matters and data is limited.
• Choose LSTM when you must run on a small device with a fixed memory budget.
• Choose a GRU first when training time is tight.
• Choose attention when you have large data and need direct links between far-apart steps.
• Choose a state-space model when sequences are very long and speed matters.
• Choose a tree model with lag features when the series is short and you need a strong, cheap baseline.

Modern forecasters such as PatchTST, N-BEATS and TFT often beat plain LSTM on benchmarks. The Temporal Fusion Transformer itself uses an LSTM layer inside, which shows the two families are not rivals so much as parts.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, columns have mixed types, many cells are missing, and column names may be absent or meaningless. See /model/architecture for how such a model is organised. The LSTM relates to this setting in several concrete ways.

Ordered rows are sequences

Many real tables are logs. Each row is an event with a timestamp, such as a machine reading, a transaction or a login. A model that predicts from one row alone throws away the history. An LSTM can read the rows of one entity in time order and carry a state that summarises the past. The final hidden state then acts as a learned feature that can be appended to a row before it enters a table model.

Feature extractor for a table model

A practical pattern is a two-stage setup. An LSTM turns each entity's history into a fixed vector. A tabular model such as TabPFN, FT-Transformer or XGBoost then uses that vector together with static columns. This separates two jobs: summarising order, and fitting the table.

Mixed types and missing values

An LSTM input is a numeric vector, so categorical columns need embeddings and numeric columns need scaling. Missing values are common in sensor tables. Good practice is to add a mask channel that marks which entries were observed, and to feed the time gap since the last observation. A foundation model for tables treats these ideas as built-in. It encodes each cell with a type-aware embedding and a missing marker, rather than relying on hand-made preprocessing. See Imputation Networks and Masked Cell Modeling.

No column names

Foundation models for tables often cannot count on useful column names. An LSTM has a similar property: it sees only positions in a vector, not names. If the order of input features is arbitrary, a recurrent network over features would be sensitive to that order. That is a reason tabular models use permutation-friendly designs such as attention across columns, as in SAINT and Axial Attention, rather than running a recurrent cell across columns.

Pre-training on synthetic or real tables

LSTMs are normally trained on one task and one dataset. Tabular foundation models differ. They are pre-trained on many tables, often synthetic ones drawn from random generators, so they can handle a new table without retraining. Synthetic sequence generators, for example random dynamical systems, can also pre-train recurrent encoders. See Synthetic Data Generation and /model/research.

In-context learning

A tabular foundation model like TabPFN reads labelled rows as context and predicts new rows in one forward pass, with no gradient steps. An LSTM does not do this by default. Its weights are fixed after training, though its hidden state does adapt as it reads a sequence. In a loose sense, a recurrent state is a small in-context memory. Attention-based tabular models make that memory explicit and much larger.

Continual learning

Recurrent networks forget old regimes when fine-tuned on new data, a trait called catastrophic forgetting. Methods such as EWC and Experience Replay are used to reduce it. For a tabular system that keeps adapting to new customers or machines, these ideas apply equally to LSTM encoders and to larger table models.

Imputation

A bidirectional LSTM is a classic imputer for time series. It reads the context before and after a gap and predicts the missing values. This is related to masked training in tabular models, where random cells are hidden and the model learns to refill them.

Anomaly detection

An LSTM trained to predict the next reading gives a natural anomaly score. A large prediction error means the new reading does not fit the learned pattern. An LSTM autoencoder, which compresses and rebuilds a window, works the same way. See Anomaly Detection and Autoencoders.

Sector and semantic classification

When a table must be classified by meaning, for example which kind of business or process it describes, the content of columns and values carries the signal. Character-level LSTMs were once used to embed column headers and cell strings. Today encoders based on attention do this better, but the idea of turning text cells into vectors for a table model is the same.

Practical guidance and best practices

Data preparation

• Fit scalers on training data only, and apply them to validation and test data.
• Keep one entity's rows together in one split.
• Use time-based splits for forecasting. Never shuffle across time.
• Add calendar features such as hour or weekday when they matter.
• Handle missing values with a mask channel plus a sensible fill.
• Clip extreme outliers before scaling when sensors can glitch.

Model design

• Start small. One or two layers with 32 to 128 hidden units is often enough.
• Add dropout between layers rather than inside the recurrence.
• Try a GRU as a faster baseline.
• Use a bidirectional LSTM only when the whole window is known at prediction time.
• Add a simple dense head, and normalise the target.

Training

• Use gradient clipping from the start.
• Use a learning rate schedule that lowers the rate when validation loss stalls.
• Stop early based on a validation split from unseen entities.
• Log both training and validation curves to catch overfitting.
• Fix random seeds, and report variation across several seeds.

Hyperparameters that matter

Tune with a search tool such as Optuna or Bayesian Optimization, and keep the validation set strictly separate from the test set.

Evaluation

• Compare against naive baselines such as "repeat the last value" and a moving average.
• Report error by horizon, because accuracy drops as you look further ahead.
• Check calibration if the model outputs probabilities or intervals.
• Test on a later time period than the training data, to mimic deployment.

Deployment

• Store the hidden and cell state per entity if you serve streaming predictions.
• Reset state when a session or machine restarts.
• Export to a portable format such as ONNX Runtime for fast inference.
• Monitor input drift. If sensor distributions shift, retrain or recalibrate.

Common mistakes

• Leaking the future. Scaling with statistics from the full dataset, or splitting windows randomly, lets information from later time into training.
• Splitting by window instead of by entity, which inflates scores because neighbouring windows are nearly identical.
• Forgetting to mask padding, so the network learns from filler values.
• Using a window that is too short to contain the signal, or so long that it adds noise and cost.
• Skipping a naive baseline, then celebrating a score that a moving average would match.
• Shuffling a single continuous stream and then carrying hidden state between batches, which breaks the meaning of the state.
• Ignoring exploding gradients, which show up as sudden NaN losses.
• Over-stacking layers on a small dataset.
• Treating an LSTM as a tabular model for unordered columns. It expects order.
• Forgetting to put the model in evaluation mode, so dropout stays active at test time.
• Feeding irregularly sampled data as if it were evenly spaced, with no time-gap feature.

Frequently asked questions

What does LSTM stand for?

LSTM stands for long short-term memory. The name describes a network whose short-term activity can be preserved over a long period through a gated memory cell. It is a kind of recurrent neural network built to learn dependencies that span many time steps.

Why is LSTM better than a plain RNN?

A plain recurrent network multiplies its signal by the same weights at every step, so learning signals tend to fade or explode over long spans. An LSTM adds a cell state that is updated by addition and controlled by gates. This keeps gradients alive across many steps and lets the network decide what to remember.

How many gates does an LSTM have?

A standard LSTM cell has three gates: forget, input and output. It also has a candidate layer that proposes new memory content. Some descriptions count four learned transformations in total, because the candidate layer has its own weights.

Is LSTM still used now that transformers exist?

Yes. LSTMs remain common for streaming inference, small datasets, embedded devices and industrial sensor tasks. Transformers dominate large language and vision work, but they need more data and compute, and they process a growing context rather than a compact state.

What is the difference between LSTM and GRU?

A GRU merges the forget and input gates into one update gate and drops the separate cell state. It has fewer parameters and trains faster. LSTM has more control through its separate cell state and output gate. Accuracy is often similar, so many teams try both, as described on the GRU page.

Can LSTM handle tabular data?

It can when rows form an ordered sequence per entity, such as event logs or sensor streams. It is not a good fit for a single unordered table with independent rows, where gradient boosting or a tabular foundation model like TabPFN is usually better.

How long a sequence can an LSTM remember?

In practice an LSTM can use dependencies across tens to a few hundred steps, depending on the task and training. Beyond that, the signal fades and other designs such as attention or state-space models tend to work better. There is no fixed limit, since memory depends on what the gates learn.

How do I use LSTM for anomaly detection?

Train the network to predict the next reading, or to reconstruct a window, using mostly normal data. At test time, a large error means the new data does not match learned behaviour. You then set a threshold from validation data. Related methods are listed under Anomaly Detection.

Do I need a GPU to train an LSTM?

Not always. Small LSTMs on modest data train on a CPU. A GPU helps for large batches, long sequences or many layers, and optimised kernels speed up the recurrence. Because steps are sequential, the speedup is smaller than for attention models.

How does LSTM relate to tabular foundation models?

An LSTM can summarise ordered row histories into features for a table model, and it shares ideas such as masking, imputation and anomaly scoring. Tabular foundation models go further by pre-training on many tables and learning in context. See /model/architecture for how that works.

Glossary

• Activation function: a function such as sigmoid or tanh applied to a layer's output.
• Backpropagation through time: gradient training that unrolls a recurrent network over its steps.
• Bidirectional: reading a sequence in both directions and combining the results.
• Candidate state: new content proposed for storage in the cell.
• Cell state: the long-term memory vector inside an LSTM.
• Constant error carousel: the near-unit gradient path that lets error signals travel far back in time.
• Dropout: randomly zeroing units during training to reduce overfitting.
• Exploding gradient: gradients that grow without bound during backpropagation.
• Forget gate: the gate that scales down old memory.
• Gradient clipping: capping gradient size to keep training stable.
• Hidden state: the output vector of the cell at each step.
• Input gate: the gate that controls how much new content is stored.
• Mask: a flag that marks padded or missing entries.
• Output gate: the gate that controls what memory is exposed.
• Recurrent neural network: a network that reuses the same weights at every time step.
• Sigmoid: a function that squashes values into the range 0 to 1.
• Tanh: a function that squashes values into the range -1 to 1.
• Truncated BPTT: backpropagation limited to fixed-length windows.
• Vanishing gradient: gradients that shrink to nearly zero over many steps.

Summary

LSTM is a gated recurrent network that keeps a long-term cell state and edits it with forget, input and output gates. This design lets gradients survive many time steps, so the network can learn long-range patterns in sensor streams, text and event logs. It trains sequentially, which makes it slower than attention models on large data, but it is compact, streams well and works on small datasets.

For tabular work, an LSTM is most useful when rows are ordered events. It can summarise each entity's history for a downstream table model, impute gaps, and score anomalies by prediction error. Tabular foundation models take related ideas further by pre-training on many tables and learning in context, as covered in /model/architecture, /model/research and /model/benchmarks. Pick LSTM for ordered, memory-driven problems with limited data, compare it with a GRU and simple baselines, and always split by entity and by time.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: