PatchTST

Overview

PatchTST is a transformer model for time-series forecasting that makes two simple design choices. First, it cuts each series into short overlapping or adjacent segments called patches, and treats each patch as one token, much like a word piece in a language model. Second, it processes every variable (channel) of a multivariate series on its own with shared weights, a design called channel independence. Together these choices make the model faster, more accurate on long horizons and easier to train than earlier transformer forecasters.

The name is short for "patch time series transformer." The idea is easy to state: a single time step carries little meaning, but a patch of several steps carries a shape, such as a rise, a dip or a short cycle. Feeding shapes to attention works better than feeding single points. If you need to predict far into the future on multivariate data such as electricity load, traffic or weather, PatchTST is a strong, widely used baseline. It sits beside N-BEATS, TFT and the pre-trained Chronos family, and it is a useful step up from simple calendar models like Prophet.

What it is

PatchTST is an encoder-only transformer applied to time series. It takes a window of past values for each channel and predicts the next H values for that channel. It does not use a decoder, and it does not need extra covariates in its basic form.

The two key ideas

1. Patching. Split a series of length L into patches of length P with a stride S. If stride equals patch length, patches do not overlap. If stride is smaller, they overlap. Each patch is embedded into a vector and becomes a token.
2. Channel independence. A multivariate series has M channels (for example, M sensors). Instead of mixing channels in one big vector per time step, PatchTST treats each channel as its own univariate series and passes them through the same transformer with shared weights. The outputs are then put back together.

Why these help

• Fewer tokens. If a window of 336 steps is cut into patches of 16 with stride 8, you get about 41 tokens instead of 336. Attention cost grows with the square of the token count, so this is a large saving. It allows longer lookbacks for the same compute.
• Local meaning. A patch holds local structure, which attention can compare across the window.
• Less overfitting. Channel independence cuts the number of parameters that tie channels together, which often reduces overfitting when channels differ.
• Reusable weights. The same transformer sees many channels, so it learns general patterns.

Everyday analogy

Reading a long text one letter at a time is slow and hard to follow. Reading by words is better. Reading by short phrases is better still. PatchTST reads time series by short phrases (patches). It also reads each sensor's story separately, as if each channel were its own short book, but with one shared reading skill.

Two training modes

• Supervised. Train directly to forecast the target horizon.
• Self-supervised. Pre-train by masking some patches and asking the model to rebuild them, then fine-tune for forecasting. The masked-patch idea is borrowed from masked language and image modeling.

Where it came from

PatchTST was introduced in a paper by Yuqi Nie and co-authors, first released as a preprint in late 2022 and presented at a major machine learning conference in 2023. The title is "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers," a play on the title of a well-known image transformer paper about image patches being worth 16 by 16 words.

The background

Transformers took over language and vision, so researchers tried them on time series. A line of models appeared that changed attention to be cheaper or to fit time series better. Many reported gains on long-horizon benchmarks. Then a short paper showed that a very simple linear model, with no attention at all, could match or beat many of these transformers on the same benchmarks. That result raised doubts about whether transformers were useful for forecasting.

The response

PatchTST answered that challenge. Its authors argued that the problem was not the transformer itself but the way it was fed. Point-wise tokens carry little information, and mixing channels at each step adds noise. By using patches and channel independence, a vanilla transformer encoder could beat the linear baselines and the earlier specialized transformers on standard long-horizon datasets, such as electricity, traffic, weather and illness data.

Related ideas it borrowed

• Patching in vision. Image transformers cut images into patches.
• Masked modeling. Masked language and masked image models pre-train by rebuilding hidden pieces.
• Instance normalization. A reversible normalization that removes the level and scale of each window before the model and restores them afterward, helping with distribution shift. A commonly used version is called RevIN.
• Shared weights across channels. A parameter-saving trick common in many sequence models.

What came after

Patching became a popular idea in time-series modeling. Several time-series foundation models use patches as tokens, and many later forecasting papers compare themselves with PatchTST. It also inspired debates on when channel independence helps and when channel mixing is needed, and designs that mix both.

How it works

This section walks through the model from raw input to forecast.

Step 1: Take a window for each channel

Suppose you have M channels and a lookback of L steps. For each channel, the input is a vector of length L.

Analogy: a weather station with several sensors. Each sensor has a strip chart of the last L readings. We handle each strip separately.

Step 2: Normalize each channel window

Each channel window is normalized by its own mean and standard deviation. This removes the level and scale so that the model focuses on shape. After the model predicts, the same mean and scale are put back. This reversible step is a major help when the level of a series drifts over time.

Step 3: Cut into patches

Choose a patch length P and a stride S. Slice the window into patches. Optionally pad the end by repeating the last value, so that the final patch is complete.

Example: L = 336, P = 16, S = 8. The number of patches is about (336 - 16) / 8 + 1 = 41, plus one if padding is used. That is the "64 words" idea: a few dozen tokens summarize hundreds of steps.

Analogy: cutting a long ribbon into short pieces with a bit of overlap so no feature is lost at the cut.

Step 4: Embed each patch

Each patch of P numbers is mapped by a linear layer into a vector of size D (the model dimension). Then a learned or fixed positional encoding is added so that the model knows the order of the patches.

Step 5: Run the transformer encoder

The sequence of patch tokens goes through several transformer encoder layers. Each layer has:

• Multi-head self-attention. Every patch looks at every other patch and decides how much to attend to each. This lets the model link a patch now with a similar patch a week ago.
• Feed-forward network. A small network applied to each token.
• Residual connections and normalization. These keep training stable.

Attention here is standard. The novelty is in what the tokens are, not in the attention rule. Faster attention kernels such as FlashAttention can be used for long windows.

Step 6: Flatten and project to the horizon

The encoder outputs one vector per patch. These are flattened into one long vector and passed through a linear head that outputs H forecast values for that channel. Each channel uses the same encoder weights and (in the main setup) the same head.

Step 7: Denormalize

Put the saved mean and scale back to return to the original units.

Step 8: Compute the loss and train

The loss is usually mean squared error (or mean absolute error) between the forecast and the true future values, averaged over channels and steps. Training uses standard optimizers such as Adam, a learning rate schedule, dropout and early stopping.

Step 9 (optional): Self-supervised pre-training

In the pre-training mode:

1. Randomly mask a share of patches in the input (for example, a large fraction) so they are hidden or set to zero.
2. Feed the masked sequence through the encoder.
3. Use a head that tries to rebuild the original values of the masked patches.
4. Use reconstruction error as the loss.

After this, the pre-trained encoder can be fine-tuned for forecasting, or used with a frozen encoder and a trained linear head. This helps when labeled forecast targets are limited, and it supports transfer between datasets with the same patch length.

Step 10: Forecast

At inference, feed the latest window for each channel, pass through the steps above and read the forecast. All horizon steps are produced at once (direct multi-step output).

Why channel independence works (and when it does not)

With channel independence, one channel's forecast uses only that channel's own history. That sounds like a loss of information. Yet in many benchmarks it wins, for three reasons. It reduces the number of parameters tying channels together. It lets the same encoder train on M times more sequences, so it sees more data. It avoids learning spurious links between channels that look related by chance in a limited history.

It can lose when channels truly depend on each other, such as a sensor that leads another by a fixed delay. For those cases, some later designs add light channel mixing, or mix channels at the head or in a few layers.

Cost and scale

• With N patches, attention cost scales as N^2.
• Using patches cuts N by the stride factor, so cost falls by about the square of that factor compared with point-wise tokens.
• Memory use also falls, so lookbacks of several hundred steps are feasible.

Key concepts and terms

A worked example

This example builds a small PatchTST by hand in PyTorch to show the logic, then shows the library route. All names are placeholders.

Step one: understand the shapes

Assume a batch of 32 windows, 7 channels, a lookback of 336 and a horizon of 96.

input shape:      [32, 7, 336]        (batch, channels, time)
after patching:   [32, 7, 41, 16]     (batch, channels, patches, patch_len)
merge channels:   [224, 41, 16]       (batch x channels, patches, patch_len)
embedding:        [224, 41, 128]      (D = 128)
encoder output:   [224, 41, 128]
flatten:          [224, 5248]         (41 x 128)
linear head:      [224, 96]
reshape back:     [32, 7, 96]

The key step is "merge channels": the batch and channel dimensions are combined, which is how all channels share the same encoder.

Step two: a minimal model

import torch
import torch.nn as nn

class TinyPatchTST(nn.Module):
    def __init__(self, lookback=336, horizon=96, patch_len=16, stride=8,
                 d_model=128, n_heads=4, n_layers=3, dropout=0.1):
        super().__init__()
        self.patch_len, self.stride = patch_len, stride
        self.n_patches = (lookback - patch_len) // stride + 1
        self.embed = nn.Linear(patch_len, d_model)
        self.pos = nn.Parameter(torch.zeros(1, self.n_patches, d_model))
        layer = nn.TransformerEncoderLayer(
            d_model, n_heads, dim_feedforward=4 * d_model,
            dropout=dropout, batch_first=True, norm_first=True)
        self.encoder = nn.TransformerEncoder(layer, n_layers)
        self.head = nn.Linear(self.n_patches * d_model, horizon)

    def forward(self, x):                       # x: [B, C, L]
        B, C, L = x.shape
        mean = x.mean(-1, keepdim=True)
        std = x.std(-1, keepdim=True) + 1e-5
        x = (x - mean) / std                    # instance normalization

        x = x.unfold(-1, self.patch_len, self.stride)   # [B, C, N, P]
        x = x.reshape(B * C, self.n_patches, self.patch_len)
        z = self.embed(x) + self.pos            # [B*C, N, D]
        z = self.encoder(z)
        z = z.flatten(1)                        # [B*C, N*D]
        y = self.head(z).reshape(B, C, -1)      # [B, C, H]
        return y * std + mean                   # denormalize

This is a teaching version. It shows normalization, patching, shared encoder and the linear head in under 30 lines.

Step three: train

model = TinyPatchTST()
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)
loss_fn = nn.MSELoss()

for epoch in range(20):
    model.train()
    for xb, yb in train_loader:          # xb: [B, C, 336], yb: [B, C, 96]
        opt.zero_grad()
        pred = model(xb)
        loss = loss_fn(pred, yb)
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
    # validate in time order, keep the best checkpoint

Use a training loop with a validation split taken later in time than the training data.

Step four: the library route

Several open-source toolkits ship a ready model. A typical call looks like this:

from neuralforecast import NeuralForecast
from neuralforecast.models import PatchTST

model = PatchTST(
    h=96,
    input_size=336,
    patch_len=16,
    stride=8,
    encoder_layers=3,
    n_heads=4,
    hidden_size=128,
    max_steps=1000,
    scaler_type="revin",
)
nf = NeuralForecast(models=[model], freq="H")
nf.fit(df=long_df)          # columns: unique_id, ds, y
pred = nf.predict()

Option names differ by library and version, so check the docs for the one you use.

A tiny numeric patching example

Take a series [1, 2, 3, 4, 5, 6, 7, 8], patch length 4 and stride 2.

patch 1: [1, 2, 3, 4]
patch 2: [3, 4, 5, 6]
patch 3: [5, 6, 7, 8]

Eight tokens would become three. Each patch is embedded into a vector, and attention compares the three vectors instead of eight single numbers. With stride 4, you would get two non-overlapping patches. The overlap in the first case helps the model see patterns that cross patch borders.

Choosing patch length and stride

A patch should span a meaningful short motif, such as part of a daily cycle. For hourly data with a 24-hour cycle, patches of 8 to 24 steps are common starting points.

Backtesting

cv = nf.cross_validation(df=long_df, n_windows=5, step_size=96)
mae = (cv["y"] - cv["PatchTST"]).abs().mean()
print("MAE over 5 windows:", float(mae))

Compare to a seasonal naive forecast and to a simple linear model. If PatchTST does not beat the linear model, the problem may not need a transformer.

Strengths and limitations

Strengths

• Strong on long horizons. Patching cuts cost and makes long lookbacks practical.
• Simple design. A standard transformer encoder with a clever input step.
• Good use of data. Channel independence turns M channels into M times more training sequences.
• Fewer parameters tied to channel count. The model size does not grow with the number of channels.
• Handles distribution shift better with instance normalization. The model sees shape, not level.
• Self-supervised option. Pre-training with masked patches helps when labels are limited and supports transfer.
• Fast inference. One forward pass yields all horizon steps.

Limitations

• Ignores cross-channel links. If channels depend on each other, independence loses information.
• No native covariates. Known future inputs, static features and categories need extensions.
• Point forecasts by default. Intervals need quantile heads, ensembles or conformal methods.
• Fixed lookback and patch layout. Changing them means retraining.
• Needs enough data. On very small sets, simple models are safer.
• Less interpretable. Attention maps exist, but there is no built-in variable importance like in TFT.
• Sensitive to settings. Patch length, stride, lookback and normalization can change results a lot.
• Gains vary by dataset. On short-horizon or highly irregular data, the advantage over linear or tree models can shrink.

How it compares with alternatives

A useful point: a plain linear model on the lookback window is a surprisingly strong baseline on long-horizon benchmarks. Always include it. PatchTST earns its place by beating it, and it often does.

Role in tabular foundation models

PatchTST is a time-series model, not a table model, but several of its ideas line up with how tabular foundation models are built. It is a good case study in how tokenization choices decide whether a transformer works.

Tokens from rows, columns and patches

A tabular foundation model must decide what a token is. Options include a cell, a column, a row or a patch of rows. PatchTST makes a clear choice for series: a token is a short run of consecutive values from one channel. This is the same spirit as column-wise tokenization in table transformers such as FT-Transformer and SAINT, where each column becomes a token. See also data tokenization for the general idea of turning raw values into model-friendly units.

Channel independence and column handling

In a table with many numeric columns, one can mix all columns in one vector (like a standard MLP) or treat each column with shared machinery and then combine. Channel independence is the second approach. Table transformers lean the same way: embed each column, then let attention decide how columns interact. The lesson from PatchTST is that forced mixing is not always good. Shared weights across columns can generalize better than a big dense layer tied to a fixed column count. That property also helps with tables whose number of columns varies from one dataset to the next, a key requirement for a foundation model that must work across many tables.

Mixed types

PatchTST takes numeric channels only. It has no embeddings for categories. For mixed tables, you would convert categories to numbers or embeddings first, or use a table-native model such as TabPFN or TabICL that handles types in one network. A hybrid design could use patch encoders for the time-based numeric block and table encoders for the rest.

Missing values

The model needs a complete grid. Missing values are usually interpolated before patching. A masked-patch pre-training scheme, though, is close to how a model can learn to handle gaps: the network is trained to rebuild hidden pieces from visible ones. That is the same skill needed for imputation, and it is one reason masked pre-training appears in so many foundation model recipes. See imputation networks and denoising autoencoders.

No column names

PatchTST uses channel position only. It never reads a name. This is the same name-blind property that matters when tabular foundation models are tested on anonymized columns: if the model works without names, it is using the structure of the values.

Pre-training on synthetic or real data

PatchTST supports self-supervised pre-training on real series, and the pre-trained encoder can be transferred to other datasets. Large time-series foundation models extend this: pre-train on a huge mix of real and generated series, often with synthetic data made from combinations of trends, cycles and noise (see synthetic data generation), then use the model zero-shot or with light tuning. Tabular foundation models like TabPFN follow a similar principle, using synthetic tables drawn from many random data-generating processes. The shared lesson is that broad, varied pre-training data can substitute for per-dataset training.

In-context and continual learning

PatchTST does not learn in context. It adapts through fine-tuning. For continual learning, teams fine-tune on recent windows with a small learning rate, or retrain periodically. Forgetting is a risk, so a mix of old and new data is typical. In-context learners instead condition on example rows or series given in the input, with no weight updates. See few-shot learning and transfer learning.

Imputation

Beyond forecasting, a masked-patch encoder can fill interior gaps: mask the missing patches and let the model reconstruct them from context. This is a natural use of the self-supervised mode. For general tables with scattered missing cells across many columns, a table-native imputer is usually a better match.

Anomaly detection

A patch-based model can flag anomalies by reconstruction or forecast error. If a patch is poorly rebuilt or poorly predicted, it is unusual. Attention maps can show which earlier patches the model compared it with. For deployment ideas see anomaly detection and predictive maintenance.

Sector and semantic classification

Patch embeddings from the encoder can be pooled into a series-level vector, and a small classifier on top can label the series: which sensor type, which sector, which machine state. This is a common way to reuse a pre-trained time-series encoder. A tabular foundation model plays a similar role at the column or table level, inferring meaning from the values.

More on how such systems are designed and tested is at /model/architecture, /model/research and /model/benchmarks.

The bottom line

PatchTST shows that the choice of token and the handling of channels can matter as much as the size of the network. Those same questions sit at the heart of tabular foundation models.

Practical guidance and best practices

Data

• Use a consistent sampling rate across all channels.
• Fill gaps before windowing and keep a record of what was filled.
• Split by time. Use the last portion for validation and the final portion for test.
• Check that the lookback covers at least one or two main cycles.
• Keep channels on a comparable footing, or rely on instance normalization.

Model settings

• Start with patch length 16 and stride 8, as in common examples, then adjust to the data.
• Try lookbacks from two to seven times the horizon.
• Use a small to moderate model: 3 layers, 128 dimensions, 4 to 16 heads.
• Use dropout and weight decay on small data.
• Turn on instance normalization.

Channel independence versus mixing

• Start with channel independence. It is simple and strong.
• If a validation study shows that channels lead and lag each other, test a variant with channel mixing.
• If you have few channels with strong coupling, a multivariate model may win.

Training

• Use AdamW with a warm-up and decaying learning rate.
• Use early stopping on a time-ordered validation split.
• Use mixed precision on a GPU to speed training.
• Try self-supervised pre-training when you have lots of unlabeled series and few labels.
• Fix seeds and repeat runs to see variance.

Evaluation

• Backtest over several windows.
• Compare against seasonal naive, a linear model and a tree model on lag features.
• Report error by horizon.
• For risk-sensitive uses, add intervals using quantile heads or conformal methods and check coverage.

Deployment

• Ship the normalization with the model.
• Fix input length. Pad or reject short windows.
• Export through a portable format such as ONNX Runtime if needed.
• Monitor drift and retrain on a schedule.
• Keep a simple fallback forecast for bad inputs.

Common mistakes

1. Skipping the linear baseline. Many long-horizon gains vanish against a well-tuned linear model.
2. Leaking the future. Normalizing with statistics from the test period, or shuffling windows across time.
3. Too long a lookback. It adds noise and cost without gains.
4. Wrong patch size. Patches much longer than the main cycle blur it. Patches that are too short act like point tokens.
5. Ignoring channel dependence. Assuming independence when channels truly lead each other.
6. Skipping normalization. Level shifts hurt a model that was trained on a different level.
7. Mismatched scaling at serve time. The most common cause of unexplained production errors.
8. Evaluating on one window. A single backtest window can be lucky.
9. Treating attention maps as explanations. They show comparisons the model made, not causes.
10. Using it on tiny data. Small datasets favor simpler models.
11. No uncertainty. Point forecasts alone hide risk.
12. Forgetting retraining. Patterns drift over time.

Frequently asked questions

What is PatchTST used for?

It is used for forecasting multivariate time series, especially over long horizons. Typical examples are electricity load, traffic, weather and sensor streams. It is also used as a pre-trained encoder for tasks such as classification and anomaly detection.

What is a patch in PatchTST?

A patch is a short run of consecutive time steps from one channel, for example 16 steps. The model turns each patch into a single vector, treating it like a word. Patches may overlap when the stride is smaller than the patch length.

What does channel independence mean?

It means each channel of a multivariate series is processed on its own by the same shared network. The model does not mix channels at each step. This keeps the model small, lets it train on more sequences, and often improves accuracy, though it can miss relationships between channels.

Why not just use point-wise tokens like earlier transformers?

A single time step has little meaning by itself, and many tokens make attention costly. Patches carry local shape and reduce the token count, which cuts compute and lets the model look further back. The authors showed that this change alone made a plain transformer competitive with strong linear baselines.

Does PatchTST work for univariate series?

Yes. A univariate series is just one channel. The model still patches the window and applies the transformer. Gains over simpler univariate models depend on how much data you have and how long the horizon is.

Can PatchTST use covariates like holidays?

The basic model does not. You can extend it by adding extra channels, by adding embeddings for known future inputs, or by using a model built for covariates, such as TFT. Test whether the added complexity improves a backtest.

Is PatchTST a foundation model?

Not by itself. It can be pre-trained with masked patches on a collection of series and then transferred, but it is not a large general-purpose model like the time-series foundation models trained on enormous mixed corpora. It does share ideas with them, especially patching.

How do I get prediction intervals?

Replace the point output with quantile heads trained with a pinball loss, train an ensemble and use the spread, or apply conformal prediction using backtest residuals. After any of these, check coverage on a held-out period.

How is it different from N-BEATS?

N-BEATS uses fully connected blocks over the whole window and focuses on univariate series. PatchTST uses a transformer over patch tokens and is built for multivariate data with shared weights across channels. N-BEATS adds an interpretable decomposition option, while PatchTST emphasizes long-horizon accuracy.

What patch length and stride should I start with?

A patch length of 16 with a stride of 8 is a common starting point for hourly or similar data. Adjust so that a patch spans a meaningful short motif of your series, and verify with a validation study rather than by rule.

Glossary

• Attention. A mechanism that lets each token weigh all other tokens.
• Channel. One variable in a multivariate series.
• Channel independence. Processing channels separately with shared weights.
• Encoder-only transformer. A transformer that has no decoder and outputs one vector per input token.
• Horizon. The number of future steps to predict.
• Instance normalization. Per-window normalization that is reversed after prediction.
• Lookback. The number of past steps used as input.
• Masked modeling. Training by rebuilding hidden parts of the input.
• Patch. A short segment of a series used as one token.
• Positional encoding. A signal that tells the model where each token sits.
• Self-supervised learning. Learning from the data itself, without labels.
• Stride. The step size between the starts of consecutive patches.
• Token. A unit of input to a transformer.
• Transformer. A neural network built around attention. See Transformers.
• Zero-shot. Predicting on new data without any training on it.

Summary

PatchTST makes a plain transformer work well for forecasting by changing how a series is fed in. It cuts each channel into patches, which shortens the sequence and gives each token a local shape, and it processes channels independently with shared weights, which saves parameters and adds training data. With instance normalization and an optional masked-patch pre-training stage, it is a strong baseline for long-horizon multivariate forecasting.

Its limits are the lack of native covariates, point outputs by default, weaker interpretability and blindness to cross-channel links. For tabular foundation models, PatchTST is a clear lesson in tokenization, shared column handling, name-blind inputs and reconstruction-based pre-training that supports imputation and anomaly detection. Test it against a linear model, a boosted tree and Chronos, keep time-ordered validation, and add TFT or a table-native model when covariates and mixed types matter.

Related

It belongs to Forecasting and tasks in my stack, under Foundation models and applied ML. Others in the same group: