TFT

Overview

The Temporal Fusion Transformer, usually shortened to TFT, is a deep learning model for multi-horizon time-series forecasting. It predicts many steps ahead at once and can use three kinds of inputs: past values of the target and other observed signals, future values that are known in advance (such as holidays or planned prices), and static facts about each series (such as a store type or region). It outputs a range of quantiles, so you get a forecast band rather than one line.

What sets TFT apart is that it tries to be both accurate and explainable. It has a built-in mechanism that scores how useful each input variable is, and its attention layer shows which past time steps mattered. That makes it a favorite when a forecast has many covariates and when a human needs to understand why. If your problem is a single clean series, simpler tools such as Prophet or N-BEATS may be enough. If you have many related series and rich side information, TFT is one of the most practical deep models to try, alongside PatchTST and the pre-trained Chronos family.

What it is

TFT is a neural network architecture that mixes recurrent layers, attention and gating. It was designed around the messy reality of business forecasting data, where inputs come in different types and have different availability.

The three types of input

Most older models either ignore this split or treat all inputs the same way. TFT handles each type through the proper door, so the model never peeks at future values that would not be available in real life.

Main building blocks

1. Gated Residual Network (GRN). A small unit that can pass its input straight through or apply a non-linear change, controlled by a gate. It lets the network skip layers that do not help.
2. Variable Selection Network (VSN). A module that learns a weight for each input variable at each time step, and then blends the variables by those weights.
3. Static covariate encoders. They turn static information into context vectors that steer the rest of the network.
4. Sequence-to-sequence layer. An LSTM encoder reads the past and an LSTM decoder reads the future known inputs, capturing local patterns.
5. Interpretable multi-head attention. A modified attention layer that finds long-range patterns and can be inspected.
6. Quantile output layer. Produces several quantiles (for example 10, 50 and 90 percent) for each future step.

Everyday analogy

Imagine a weather and logistics planning room. A planner starts with fixed facts about each warehouse (static). They add the calendar and the planned promotions for the coming weeks (known future). They look at past deliveries and the weather so far (observed past). A senior colleague (the variable selection) says which of these facts are worth listening to today. A memory expert (the LSTM) tracks recent local trends. A historian (attention) points to similar events weeks or months ago. Finally the planner gives not one number but a likely range. TFT combines these roles in one model.

Where it came from

TFT was introduced in a paper by Bryan Lim and co-authors. A first version appeared as a preprint in 2019, and a final journal version came out in 2021 in a major forecasting journal. The paper was titled "Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting."

The problem it addressed

By the late 2010s, deep learning had made progress in forecasting, but most models were black boxes. Business users wanted to know which variables mattered, what the model learned about seasonality, and when it noticed unusual events. At the same time, real datasets had a mix of covariates. Earlier neural forecasters often dealt poorly with the mix of static, known and observed inputs.

Design influences

• The original transformer. Attention lets a model look at any past step directly, avoiding the long chain of recurrence.
• Recurrent networks. LSTMs give a sense of local order, which attention alone does not.
• Gating and residual links. These come from deep learning practice for stabilizing and simplifying deep networks.
• Quantile regression. An old statistical idea for predicting ranges instead of single values.
• Earlier probabilistic forecasters. Models that predict distributions for many related series paved the way for the global-model approach.

Reception and spread

The model was reported to improve on strong baselines across several public datasets, including electricity, traffic and retail data. It became available in open-source libraries and in some cloud forecasting services. It also inspired follow-ups that simplify or speed up the idea. Today it is a standard entry in forecasting model collections and a common reference point in comparisons, including those with PatchTST.

How it works

This section follows the data through TFT step by step.

Step 1: Organize the data

TFT is trained on a long table. Each row is one time step of one series. Columns include a series id, a time index, the target, static columns, known future columns and observed columns. The library cuts this table into windows: a past encoder window of length L and a future decoder window of length H.

Analogy: for each exam question, give the student the notes from the last L days and the schedule for the next H days, and ask for the grades.

Step 2: Embed and scale the inputs

• Categorical columns are turned into learned embeddings, which are small vectors that capture similarity between categories.
• Continuous columns are scaled, often per series.
• Each variable is projected to the same size so they can be mixed.

Step 3: Variable selection

For each time step, the VSN looks at all variables of one type and produces a weight vector that sums to one. Each variable is processed by its own small network, and the outputs are mixed using the weights.

Analogy: a mixing desk in a studio. Each instrument has its own fader. The engineer slides up the useful ones and slides down the noisy ones, and the faders move over time.

The weights are saved. Averaged over a dataset, they tell you which variables the model relies on. A feature like "day of week" may be high, while a column of random noise should be low.

Step 4: Static context

Static covariates are passed through GRNs to make four context vectors. They are used to:

• steer variable selection (so a store type changes which inputs matter),
• initialize the LSTM states,
• enrich the features after the LSTM,
• condition the attention layer.

This means that static information influences the whole network, not just one place.

Step 5: Local processing with LSTMs

The encoder LSTM reads the selected past inputs in time order. The decoder LSTM reads the selected known future inputs. Their outputs are combined with the original inputs through a gating layer, so the network can choose how much to rely on the LSTM and how much on the raw features.

This stage captures local patterns, such as a short-term trend or an immediate response to an event.

Step 6: Static enrichment

A GRN mixes the LSTM outputs with the static context. This adds series-specific knowledge to every time step.

Step 7: Interpretable multi-head attention

Attention lets each future step look at every past step and decide how much weight to give it. TFT modifies standard multi-head attention so that the heads share value weights. As a result, the average attention across heads can be read as a single set of importance scores over time.

Analogy: the historian highlights past days that look like the day we are forecasting, such as the same weekday last week or the same week last year. The highlight pattern can be plotted.

A causal mask ensures that a step cannot look at the future, only at the past and itself.

Step 8: Position-wise feed-forward and a final gate

After attention, another GRN processes each position, and a skip connection to the LSTM output keeps training stable. A final gate decides how much of the attention path to use.

Step 9: Quantile outputs

A linear layer maps each future position to several quantiles. The model is trained with the quantile loss, also called the pinball loss. For a quantile q, the loss penalizes under-prediction and over-prediction asymmetrically. Summing over quantiles and steps trains the network to give a calibrated range.

Step 10: Read the explanations

After training, you can extract three kinds of insight:

• Variable importance from the selection weights (static, past and future).
• Temporal importance from the attention weights.
• Seasonal patterns by looking at attention across a long window, which can show weekly or yearly peaks.

Why this design helps

• The split into input types prevents leakage and uses information properly.
• Gating lets the model be simple when the data is simple.
• Attention captures long-range patterns that LSTMs forget.
• LSTMs capture local order that attention might blur.
• Quantile outputs support real decisions, such as stocking to the 90th percentile.

Key concepts and terms

A worked example

This example uses a common open-source library built on PyTorch Lightning and PyTorch. Names and data are placeholders.

The data table

import pandas as pd
import numpy as np

df = pd.read_csv("store_daily.csv")      # store_id, date, sales, price, promo, temp, store_type
df["date"] = pd.to_datetime(df["date"])
df = df.sort_values(["store_id", "date"])

# integer time index per series
df["time_idx"] = (df["date"] - df["date"].min()).dt.days
df["day_of_week"] = df["date"].dt.dayofweek.astype(str)
df["month"] = df["date"].dt.month.astype(str)

Categorical columns are turned into strings so the library treats them as categories.

Declaring the roles of each column

from pytorch_forecasting import TimeSeriesDataSet
from pytorch_forecasting.data import GroupNormalizer

max_encoder = 60      # lookback
max_pred = 14         # horizon
cutoff = df["time_idx"].max() - max_pred

training = TimeSeriesDataSet(
    df[df.time_idx <= cutoff],
    time_idx="time_idx",
    target="sales",
    group_ids=["store_id"],
    min_encoder_length=max_encoder // 2,
    max_encoder_length=max_encoder,
    min_prediction_length=1,
    max_prediction_length=max_pred,
    static_categoricals=["store_id", "store_type"],
    time_varying_known_categoricals=["day_of_week", "month"],
    time_varying_known_reals=["time_idx", "price", "promo"],
    time_varying_unknown_reals=["sales", "temp"],
    target_normalizer=GroupNormalizer(groups=["store_id"], transformation="softplus"),
    add_relative_time_idx=True,
    add_target_scales=True,
)
validation = TimeSeriesDataSet.from_dataset(training, df, predict=True, stop_randomization=True)

This is where the three input types become real. Price and promo are known in the future. Temperature is only observed in the past. Store type is static.

Building and training

from pytorch_forecasting import TemporalFusionTransformer
from pytorch_forecasting.metrics import QuantileLoss
import lightning.pytorch as pl

train_loader = training.to_dataloader(train=True, batch_size=64, num_workers=2)
val_loader = validation.to_dataloader(train=False, batch_size=128, num_workers=2)

tft = TemporalFusionTransformer.from_dataset(
    training,
    learning_rate=0.03,
    hidden_size=32,
    attention_head_size=4,
    dropout=0.1,
    hidden_continuous_size=16,
    loss=QuantileLoss(),
)

trainer = pl.Trainer(max_epochs=30, accelerator="auto", gradient_clip_val=0.1)
trainer.fit(tft, train_dataloaders=train_loader, val_dataloaders=val_loader)

Typical hidden sizes are small to moderate. Larger sizes raise cost and the risk of overfitting.

Predicting and plotting

raw, x = tft.predict(val_loader, mode="raw", return_x=True)
tft.plot_prediction(x, raw, idx=0, add_loss_to_title=True)

The plot shows the median line and shaded quantile bands against the actual values.

Looking at interpretation

interpretation = tft.interpret_output(raw, reduction="sum")
tft.plot_interpretation(interpretation)

The output has bar charts for static, encoder and decoder variable importance, and a curve of attention over past steps. Suppose the encoder chart shows sales and temperature near the top and the decoder chart shows promo above price. That tells a planner that planned promotions drive the forecast more than price changes in this dataset.

A numeric sense of quantile loss

Take a quantile q = 0.9, an actual value of 100 and a prediction of 80. The error is 100 - 80 = 20, an under-prediction. The pinball loss is q x error = 0.9 x 20 = 18. If the prediction were 120 instead (over-prediction by 20), the loss would be (1 - q) x 20 = 0.1 x 20 = 2. Under-predicting is penalized nine times more, so the model learns to put the 90th percentile line above most actual values. At q = 0.5 both sides cost the same, so the line becomes the median.

Checking interval coverage

A forecast band is only useful if it is honest. After a backtest, count how often the actual value fell inside the band.

def coverage(actual, lower, upper):
    actual, lower, upper = map(np.asarray, (actual, lower, upper))
    inside = (actual >= lower) & (actual <= upper)
    return float(inside.mean())

# for a 10th-90th band, aim for roughly 0.80
print(coverage(y_true, q10, q90))

If coverage is far below the target, the model is overconfident. Reasons include too little data, a regime change, or a validation window that is not like the training window. If coverage is far above the target, the band is too wide to be useful.

What the importance charts might look like

Suppose you trained on daily store data and the charts show the following pattern.

A planner can act on this: confirm the promotion calendar early, because it moves the forecast the most. A data scientist can act on it too: if a variable has near-zero weight everywhere, remove it and retrain, which is simpler and often as accurate.

Practical tuning order

1. Fix the horizon and lookback from the business need.
2. Run with defaults and a small hidden size.
3. Tune the learning rate first. It matters most.
4. Tune hidden size, attention heads and dropout.
5. Add more covariates only if variable importance shows they help.
6. Compare with a simple baseline each time.

Strengths and limitations

Strengths

• Rich covariate support. Static, known future and observed inputs all have a place.
• Multi-horizon output. One pass yields the whole forecast window.
• Probabilistic by design. Quantile outputs give ranges.
• Interpretability. Variable importance and attention patterns are available.
• Global learning. One model learns from many related series, helping those with short history.
• Handles mixed types. Categorical embeddings and continuous values live in one model.
• Robustness from gating. The network can suppress inputs that add noise.

Limitations

• Heavy compared with simple models. Training takes longer and needs more tuning than Prophet or ARIMA.
• Needs data. Small datasets can overfit, and a simpler model may win.
• Data preparation is demanding. The three input types must be correct, and future known inputs must truly be known.
• Interpretability has limits. Attention and selection weights show correlation with the model's use of inputs, not causes in the world.
• Fixed lookback. Very long history beyond the window is lost.
• Quantile crossing. Separate quantile outputs can sometimes cross, so the 90th line dips below the 50th. Post-fixes exist.
• Not designed for very long horizons. Newer patch-based designs such as PatchTST may do better at long horizons.
• Many hyperparameters. Hidden size, heads, dropout, lookback, learning rate and loss weights all interact.

How it compares with alternatives

An honest note: on many tabular-style forecasting problems with good feature engineering, boosted trees are a tough competitor. TFT earns its place when the temporal structure is rich, when many related series help each other, and when you value the interpretation tools.

Role in tabular foundation models

TFT is not a tabular foundation model, but it is one of the clearest examples of a deep model that treats a table with mixed column types as a first-class input. That makes it a useful reference for what table-native models also need to do.

Rows and columns with roles

A table for TFT has rows (time steps) and columns that fall into types: static, known, observed, plus a target. A tabular foundation model also has to cope with columns of different roles. TFT makes the role explicit and wires it into the network. A foundation model usually infers roles from the data and treats all columns more symmetrically. Both ideas target the same problem: do not pretend all columns are alike.

Mixed types

TFT embeds categories and scales numbers, then brings both into a common space. This is the same recipe used by table transformers such as FT-Transformer and SAINT. The common theme is turning each column into a token-like vector so attention can compare columns. TFT does this inside a time-series frame. TabPFN and TabICL do it for general tables, using pre-training to remove the need for per-dataset training.

Variable selection as feature selection

The VSN is a learned feature selector. In tabular modeling, deciding which columns matter is a core step, often done with importance scores from trees or with SHAP. TFT bakes that into the model and exposes the weights. Foundation models for tables tend to handle irrelevant columns by learning, during pre-training, to ignore columns that carry no signal, including on synthetic tables where noise columns are added on purpose.

Missing values

TFT expects a complete grid in its basic form. Missing covariates are usually filled or flagged before training, and some libraries support a missing-indicator approach. Tabular foundation models are often trained with masked cells and can treat missing values natively. For fill-in methods see imputation networks.

No column names

TFT identifies columns by their declared role and by the learned embedding of each variable. The text of a column name is not used. It is therefore name-blind, which is the property tested when tabular models are evaluated on anonymized columns. What matters is the relationship between the values and the target, not the label on the header.

Pre-training on synthetic or real tables

TFT is trained from scratch on the target dataset, although the global-model approach lets it share strength across series. It is not pre-trained across unrelated datasets. Time-series foundation models like Chronos and tabular ones like TabPFN show another path: pre-train on a huge, varied pool, often with generated data (see synthetic data generation), then apply the model with little or no extra training. A realistic workflow is to use such a model as a strong first guess and TFT where rich covariates justify a custom model.

In-context and continual learning

TFT does not learn in context. To adapt to new conditions, you fine-tune or retrain on recent data. This is simple continual learning: periodic retraining with a rolling window, or fine-tuning from the last checkpoint with a small learning rate. The risk is forgetting, so teams usually keep a mix of old and new windows. In-context learners, by contrast, take a handful of labeled rows as part of the input and adapt without updating weights. See few-shot learning and transfer learning.

Imputation

Forecasting is a close cousin of imputation at the edge of a series: predicting the next values is like filling a gap that happens to lie in the future. TFT can be adapted to fill interior gaps by masking known values and training the network to predict them, but this is not its normal use. Specialized imputers and autoencoders are better suited.

Anomaly detection

Because TFT provides quantiles, an observed value outside the 10th to 90th range is a natural alert. With attention and variable weights, you can also explain why a value was unexpected, for example that the model relied heavily on a promotion flag that did not take place. See anomaly detection and predictive maintenance for related applications.

Sector and semantic classification

Static embeddings in TFT group similar series: stores, products or machines that behave alike end up close together. Those embeddings can be reused for classification or clustering, such as grouping series by sector or type. A tabular foundation model reads whole rows and columns and can predict such labels directly. TFT's embeddings are a by-product, but a useful one.

For deeper background on how these models are built and tested, see /model/architecture, /model/research and /model/benchmarks.

The bottom line

TFT shows how to respect the structure of a real table inside a deep model: typed columns, selection of variables, long-range attention and honest uncertainty. Tabular foundation models generalize those ambitions across datasets.

Practical guidance and best practices

Data

• Make sure known future inputs are really known at forecast time. A promotion plan is known. Next week's temperature is not, unless you have a forecast for it, and then you should treat the forecast as the known input.
• Keep a clean, gap-free time index per series, or let the library handle gaps explicitly.
• Add calendar features (day of week, month, holiday flags) as known inputs.
• Use series ids as static categoricals so the model can learn series-level effects.
• Split by time, never randomly.

Model setup

• Start small: hidden size of 16 to 64, a few attention heads, moderate dropout.
• Use a lookback that covers at least one seasonal cycle.
• Use quantiles that match decisions, for example 10, 50 and 90.
• Normalize targets per series so series of different size train together.

Training

• Tune the learning rate first, using a range test if the library provides one.
• Use gradient clipping.
• Use early stopping on validation loss.
• Train on a GPU for large datasets.
• Fix the random seed and keep a record of settings.

Evaluation

• Backtest over several cutoffs.
• Report the median forecast error and the interval coverage.
• Compare with seasonal naive and with a boosted tree on lag features.
• Look at the worst series and the worst horizons.

Interpretation

• Average variable weights over the validation set, not just one example.
• Treat unusual weights as hypotheses to check with domain experts.
• Remember that attention is not proof of cause.

Deployment

• Freeze the data pipeline code with the model so that encoders and scalers match.
• Export the model with care. The library may need its own serialization. Some teams export parts to ONNX Runtime.
• Monitor error and coverage over time.
• Retrain on a schedule, or when drift is detected.

Common mistakes

1. Leaking future information. Using a variable as "known" when its future value would not be available.
2. Random splits. They inflate accuracy and hide the true error.
3. Too many covariates for too little data. The model overfits noise.
4. Ignoring the baseline. TFT may not beat a boosted tree with good features.
5. Reading attention as causal. It shows what the model used, not what drives the world.
6. Different scaling at train and serve time. Predictions drift for no obvious reason.
7. Lookback too short. The model cannot see weekly or yearly cycles.
8. Not checking coverage. An 80 percent band that covers 50 percent of actual values is misleading.
9. Overlapping validation windows. They give optimistic error estimates.
10. Using one huge hidden size from the start. Training becomes slow and unstable.
11. Wrong category handling. Unseen categories at inference time can cause errors unless handled.
12. Neglecting retraining. Old models decay as behavior shifts.

Frequently asked questions

What is the Temporal Fusion Transformer used for?

It is used for multi-horizon forecasting when you have many covariates and many related series. Typical cases include retail demand, energy load, traffic volume and inventory planning. It is chosen when you need quantile forecasts and want to see which inputs mattered.

Why is it called a fusion model?

It fuses several kinds of information: static attributes, known future inputs and observed past signals. It also fuses two ways of reading time, local recurrent processing and long-range attention. The name points to this blending of inputs and mechanisms in one network.

Is TFT really interpretable?

It is more inspectable than most deep forecasters. You can read variable importance weights and attention patterns, which are useful for debugging and for discussion with experts. They show how the model used its inputs, though, and not why the real world behaves as it does, so they should be treated as clues rather than proof.

How many series do I need to train a TFT?

There is no fixed number, but it does best with many related series or a long history that yields thousands of windows. With a handful of short series, simpler models are safer. The global training approach helps series with little history borrow strength from others.

Can TFT handle missing data?

The basic setup expects a regular time grid with values present. Gaps in the target are typically filled or the windows that contain them are skipped. Missing covariates should be imputed or flagged before training. Some libraries offer options, so check the version you use.

Does TFT produce a single forecast or a range?

It produces a range. The output layer predicts several quantiles for every future step, and the median acts as the central forecast. You pick the quantiles that match your decisions, such as the 90th percentile for safety stock.

How does TFT differ from a plain transformer?

A plain transformer applies attention to all inputs treated alike. TFT adds typed inputs, variable selection, gating, recurrent layers for local order, a modified attention for interpretation and quantile outputs. Those parts make it better suited to the structure of forecasting data.

Is TFT better than gradient boosting for forecasting?

Sometimes, but not always. Boosted trees with strong lag and calendar features are hard to beat on many problems and are faster to train. TFT tends to help when you have many related series, rich covariates of different types and a need for joint multi-horizon quantile outputs.

Can I use TFT with a pre-trained foundation model?

Yes, in a combined workflow. A pre-trained model can give a zero-shot forecast that serves as a baseline or as an extra input feature. TFT can then learn corrections from covariates that the foundation model does not see. Always check by backtest that the combination helps.

What hardware does TFT need?

Small problems run on a CPU. For hundreds or thousands of series with long windows, a single GPU makes training much faster. Memory use depends on batch size, hidden size and window length, so reduce those first if you run out of memory.

Glossary

• Attention. A mechanism that weights past positions by relevance.
• Categorical embedding. A learned vector that represents a category.
• Covariate. An extra variable that helps predict the target.
• Decoder. The part of the network that handles the forecast window.
• Encoder. The part of the network that reads the history.
• Gated Linear Unit. A gate that scales a signal between zero and its full value.
• Gated Residual Network. A block that mixes a skip path with a gated transformation.
• Horizon. The number of future steps to predict.
• Known future input. A variable whose future values are available at forecast time.
• LSTM. A recurrent network that keeps a memory of the past.
• Observed input. A variable known only for past time steps.
• Pinball loss. The loss used to train quantile predictions.
• Prediction interval. A range expected to contain the future value.
• Quantile. A cut point such that a given share of values lie below it.
• Static covariate. An attribute that does not change over time.
• Variable selection. Learning how much weight each input variable should get.

Summary

The Temporal Fusion Transformer is a deep forecasting model designed for real, covariate-rich data. It sorts inputs into static, known future and observed past, selects useful variables with learned weights, reads local order with LSTMs, finds long-range patterns with interpretable attention and outputs quantile forecasts for several steps ahead.

Its strengths are accuracy on complex problems, probabilistic output and built-in explanation tools. Its costs are tuning effort, data preparation and a need for enough data. For tabular foundation models, TFT is a clear example of how a deep network can respect typed columns, choose features and stay name-blind, while models such as TabPFN, TabICL and Chronos aim to remove per-dataset training through large-scale pre-training. Try a simple baseline first, add TFT when covariates and multiple horizons justify it, and always validate by backtesting in time order.

Related

It belongs to Forecasting and tasks in my stack, under Foundation models and applied ML. Others in the same group: