Titans and MIRAS: test-time memory beyond the Transformer

Why people say "Transformer 2.0"

Since 2017 almost every large language model has been built on the Transformer. It is powerful, but the longer the input, the more expensive it becomes. Doubling the length of a document roughly quadruples the work done by attention.

Between late 2024 and late 2025, a group at Google Research published a series of papers that attack this weakness from an unusual angle. Instead of making attention cheaper, they gave the model a second kind of memory: a small neural network that keeps learning while the model reads. The papers are Titans, MIRAS, ATLAS and Nested Learning, all with Ali Behrouz as first author and Vahab Mirrokni as senior author.

"Transformer 2.0" is a nickname, not an official name. Google does not use it in the papers or on its research blog. A widely shared YouTube video by AI commentator Matthew Berman was titled "Google Research Unveils 'Transformers 2.0' aka TITANS", and in January 2026 a developer blog post by Fotie M. Constant ran the headline "Google Just Dropped Transformer 2.0: Meet 'Nested Learning'". Do not confuse any of this with Sakana AI's "Transformer²", a separate January 2025 paper on self-adaptive language models.

How a standard Transformer remembers

Attention is a precise short-term memory

The attention mechanism lets every token look at every earlier token and decide how much each one matters. It is like an open-book exam: you can flip back to any page at any moment, and nothing is forgotten.

The price is that the book has to stay open on the desk. During generation the model keeps the keys and values of every past token in a KV cache, which grows with every token, while attention during training grows with the square of the sequence length. At some point the desk is full. This is why context windows are finite and very long contexts are slow and costly.

Recurrent models carry a fixed-size notebook

The alternative family is recurrent neural networks and their modern descendants: linear recurrent models and state-space models such as Mamba. They read one step at a time and squeeze everything into a state of fixed size.

Think of a pocket notebook with a fixed number of pages. Updating it is cheap, but a 2-million-token document must fit into the same pages as a 2-thousand-token one. Detail gets lost, and recalling one fact from far back becomes hard.

The Titans paper frames attention as short-term memory and argues for a separate long-term memory that is compact but can still hold what matters. Google's blog describes the goal as combining "the speed of RNNs with the accuracy of transformers".

How Titans memory works

Memory as a learner, not a container

In Titans the long-term memory is a small multi-layer perceptron. Its weights are the memory. When a new token arrives, the model turns it into a key and a value and asks the memory: given this key, can you produce this value? If the answer is poor, the memory takes a small gradient descent step to do better next time.

This happens at test time, while the model is reading.

An everyday picture: a new employee keeps a notes app. When something happens that they expected, they write nothing. When something surprises them, they write it down. Over time the notes hold the surprises, not the routine.

The update rule

The Titans paper (Sections 3.1 and 3.2) writes the update as follows.

k_t = x_t W_K          key made from the current input x_t
v_t = x_t W_V          value made from the same input
loss  ℓ(M_{t−1}; x_t) = || M_{t−1}(k_t) − v_t ||²

S_t = η_t · S_{t−1} − θ_t · ∇ℓ(M_{t−1}; x_t)     surprise with momentum
M_t = (1 − α_t) · M_{t−1} + S_t                  memory update with forgetting

read:  y_t = M*(q_t),  q_t = x_t W_Q             forward pass, no weight update

M_t     the memory: the weights of a small MLP after step t
x_t     the current input token (as a vector)
W_K, W_V, W_Q   learned projections for keys, values and queries
ℓ       associative memory loss: how badly the memory maps key to value
∇ℓ      gradient of that loss: the "momentary surprise"
S_t     accumulated surprise, a momentum term
η_t     data-dependent decay of past surprise
θ_t     data-dependent step size for momentary surprise
α_t     forgetting gate in [0, 1], a form of weight decay
M*      the memory used for reading, without changing its weights

Read it in three parts.

Surprise. The gradient is large when the memory's prediction is far from the true value. The blog's example: an expected word produces a small gradient, while a banana peel appearing in a financial report produces a large one and is prioritised for storage.
Momentum. A surprising token is often followed by related tokens that are not surprising on their own. The term η_t · S_{t−1} carries earlier surprise forward so the follow-up is stored too.
Forgetting. The factor (1 − α_t) shrinks the old memory a little each step. Because α_t depends on the data, the model can learn when to clear space.

Three ways to wire it in

MAC, memory as a context. The sequence is cut into segments. For each segment the model retrieves from long-term memory, prepends that with a few learned "persistent memory" tokens (task knowledge that does not depend on the input), and runs attention over the block. The output then updates the memory.
MAG, memory as a gate. Sliding-window attention and the neural memory run side by side and are mixed by a gate.
MAL, memory as a layer. The memory compresses the input first, then sliding-window attention processes the result.

To keep training fast, memory updates are computed in chunks so much of the work runs in parallel. This detail matters, as we will see.

What Titans reported

The first arXiv version (December 2024) reports models from 170M to 760M parameters and says larger-model results were in progress. The paper was presented as a poster at NeurIPS 2025. Its main claims:

Language modelling and common-sense reasoning: Titans variants outperform Transformers and modern linear recurrent models at matched sizes.
Needle in a haystack: on single-needle tasks from RULER at 2K to 16K tokens, the neural memory beats TTT, Mamba2 and DeltaNet, with MAC the strongest Titans variant.
BABILong: reasoning over facts scattered through very long text. Titans (MAC) is reported to outperform much larger models, including GPT-4, in few-shot and fine-tuned settings.
Scale: Titans can scale beyond a 2-million-token context window with higher needle-in-haystack accuracy than baselines.
DNA modelling: on GenomicsBenchmarks the memory is competitive with state-of-the-art models, winning some tasks and losing others.
Time series forecasting: covered below.

MIRAS: the general recipe

Four design choices

The second paper, "It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization" by Ali Behrouz, Meisam Razaviyayn, Peilin Zhong and Vahab Mirrokni, appeared on arXiv in April 2025 and was published at ICLR 2026. It names a framework called MIRAS. The name is unrelated to MIDAS, the denoising-autoencoder imputation method.

MIRAS treats Transformers, Titans and linear RNNs alike as associative memories, each described by four choices:

Memory architecture: a vector, a matrix, or a deep MLP as in Titans.
Attentional bias: the internal objective the memory minimises when storing a key–value pair. Most existing models use dot-product similarity or squared (L2) error.
Retention gate: how the memory balances new learning against keeping the old. MIRAS reframes forgetting as regularisation.
Memory learning algorithm: the optimiser, such as gradient descent with or without momentum.

Seen this way, Titans is an MLP memory with an L2 objective, weight-decay retention and momentum.

Moneta, Yaad and Memora

The paper builds three attention-free models, each with a two-layer MLP memory updated by gradient descent.

Moneta uses an ℓp loss (p = 3 in the experiments) and an ℓq-norm retention gate (q = 4).
Yaad uses a Huber loss, which acts like squared error for small mistakes and absolute error for large ones, making it less sensitive to outliers.
Memora keeps the L2 loss but uses a KL-divergence retention gate, so the update behaves like a softmax-normalised probability map.

The first arXiv version trains models of 120M to 1.3B parameters against Transformer++, Mamba-2 and Gated DeltaNet on language modelling, common-sense reasoning and needle-in-haystack retrieval, with different variants strongest on different tasks. It has no time series experiment; the ICLR 2026 abstract adds time series tasks to its claims.

ATLAS

"ATLAS: Learning to Optimally Memorize the Context at Test Time" (Behrouz and seven co-authors, May 2025) notes that Titans-style memories update using only the latest input, and instead optimises memory over a window of current and past tokens. It also proposes DeepTransformers, which strictly generalise the original Transformer, and reports more than 80% improvement over Titans on BABILong at a 10-million-token context. The abstract does not give the baseline score, so treat the figure as indicative.

Nested Learning and HOPE

"Nested Learning: The Illusion of Deep Learning Architectures" by Behrouz, Razaviyayn, Zhong and Mirrokni was presented at NeurIPS 2025 and posted to arXiv on 31 December 2025. Google introduced it on 7 November 2025 as "a new ML paradigm for continual learning".

The core idea is that a model is a set of nested learning problems, each updating at its own speed. The fast inner loop of Titans memory and the slow outer loop of training are two levels of the same thing. Even optimisers such as momentum or Adam can be read as associative memories that compress past gradients.

Continuum memory system. Instead of one short-term and one long-term memory, there is a spectrum of memory blocks updating at different frequencies, like a day planner, monthly goals and life habits.
HOPE. A proof-of-concept variant of Titans that can modify its own update rule and adds continuum memory blocks. The blog reports better perplexity and accuracy than Titans, Samba and a baseline Transformer, and better needle-in-haystack results than Titans, TTT and Mamba2, without giving numbers.

The motivation is catastrophic forgetting: a model learning a new task tends to lose an old one.

Time series: where test-time memory makes sense

Why drifting series need adaptive memory

Many real time series are long and non-stationary: electricity demand shifts with the seasons, traffic changes after a new road opens, a machine's vibration drifts as a bearing wears. Titans-style memory fits this in two ways. It adapts while running, taking a gradient step on each new chunk, so it can absorb a level shift without retraining. And it forgets on purpose, letting old regimes fade after a structural break.

What the Titans paper reports for forecasting

The experiment is in Section 5.6, and it is narrower than headlines suggest.

Setup: the authors took Simba, an existing Mamba-based forecasting framework, and replaced its Mamba module with the Titans neural memory.
Datasets: ETTm1, ETTm2, ETTh1, ETTh2, ECL (electricity), Traffic and Weather.
Baselines: Simba, iTransformer, RLinear, PatchTST, Crossformer, TiDE, TimesNet and DLinear.
Metrics: mean squared error and mean absolute error.
Claim: the neural memory outperforms all baselines, including Mamba-based, linear and Transformer-based models.

The table largely supports this, though not in every cell; iTransformer has a lower MAE on Traffic (0.282 against 0.289). This is a supervised model trained per dataset, not a pretrained foundation model.

Relation to time series foundation models

Time series foundation models such as Chronos are pretrained on many series and forecast new ones zero-shot from a context window. A neural long-term memory is a plausible way to condition on years of history while attending only to recent weeks. I have not found a published time series foundation model from these authors built on Titans memory, so this is an outlook, not a result.

Honest limits

Reproducibility. "Titans Revisited" by Di Nepi, Siciliano and Silvestri (arXiv, October 2025) reimplemented Titans and tested it on masked language modelling, forecasting and recommendation. The neural memory consistently helped compared with attention-only models, but Titans did not always beat established baselines, partly because of chunking. The authors also note the original work had no public code.
Compute. Every chunk triggers a backward pass through the memory: cheaper than attending to millions of tokens, dearer than a plain forward pass.
Noise. A memory that learns from surprise also learns from sensor glitches. Robust objectives help in principle; data cleaning is still needed.

ARIMA and ARMIN

ARIMA, the baseline to run first

ARIMA predicts the next value of a series from a few past values and past errors, after differencing to remove trend. Its memory is explicit and short.

Aspect	ARIMA	Titans-style neural memory
Memory	A few fixed lags and errors	MLP weights updated per chunk
Adaptation	Refit on new data	Gradient step at test time
Form	Linear	Non-linear
Data needed	One series	Large training set
Interpretability	High	Low
Cost	Tiny	GPU-scale

If a deep model cannot beat a tuned ARIMA or a seasonal naive forecast on your data, its long-term memory is not helping you.

ARMIN is real, but not a forecasting model

There is a genuine model called ARMIN: "ARMIN: Towards a More Efficient and Light-weight Recurrent Memory Network" by Zhangheng Li and co-authors at Peking University, IJCAI 2019. The name stands for Auto-addressing and Recurrent Memory Integrating Network. It keeps a matrix of past hidden states, picks one slot to read with a Gumbel-softmax, and writes the new state back into that slot. It was evaluated on algorithmic tasks such as copy and associative recall, permuted pixel MNIST and character-level language modelling, and presented as lighter than Neural Turing Machines and the Differentiable Neural Computer. It belongs to the older family of memory-augmented recurrent networks, but it does not learn at test time in the Titans sense and was not evaluated on forecasting.

Tabular foundation models already learn at test time

In-context learning on rows

Tabular foundation models such as TabPFN and TabICL take the training rows and test rows together in one forward pass and predict the test labels without any gradient update. This is in-context learning: the learning happens in the activations.

The difference from Titans is where knowledge is stored. In these models the training rows sit in the context, like the open-book exam. In Titans, past inputs are written into the weights of a small network, like the notes app. The first is exact but bounded; the second is compressed but can keep growing.

The row limit

Because rows attend to rows, cost grows quickly with the number of training rows. The TabPFN version published in Nature in January 2025 targets up to 10,000 rows and 500 features. TabICL (ICML 2025) is pretrained on synthetic datasets of up to 60,000 rows and reports handling up to 500,000 rows on affordable hardware. Larger tables are usually handled by subsampling, retrieving nearby rows, or ensembling over chunks.

Where neural memory could help (outlook)

This subsection is reasoned outlook, not published result. I am not aware of a peer-reviewed tabular foundation model using Titans or MIRAS memory as of November 2026.

Streaming rows. Transactions, sensor logs and claims grow continuously. A memory could absorb new rows chunk by chunk while attention covers the most recent rows exactly: the MAC pattern applied to rows.
Very large tables. Instead of subsampling, stream the whole table through a memory and answer queries against both memory and a small attended context.
Time-indexed tables. The forgetting gate offers a natural way to down-weight older regimes in credit, demand or fraud data.
Messy values. A Huber-style objective, as in Yaad, is an obvious fit for tables full of outliers.

The obstacles are real. Table rows are usually exchangeable, so streaming order should not change the answer, yet a sequential memory is order-sensitive by design. Test-time gradient steps make inference slower and harder to audit. And strong tabular in-context learners rely on exact comparison between a test row and similar training rows, which compression may blur. Anyone building a large tabular foundation model needs to measure these trade-offs, not assume them.

What engineers can use today

Code and maturity

As far as I can find, Google has not released official code for Titans, MIRAS, ATLAS or HOPE. The most used community implementation is titans-pytorch by Phil Wang (lucidrains), on PyPI under the MIT licence and labelled unofficial. It provides a NeuralMemory module and a MemoryAsContextTransformer.

The snippet below ran on CPU with titans-pytorch 0.6.0. It processes a sequence in two parts and carries the memory state between them, the streaming pattern discussed above.

import torch
from titans_pytorch import NeuralMemory

mem = NeuralMemory(dim=64, chunk_size=16)   # memory updated every 16 steps

first = torch.randn(1, 256, 64)             # batch, sequence, features
second = torch.randn(1, 256, 64)

out1, state = mem(first)                    # read and write memory
out2, state = mem(second, state=state)      # continue from stored memory

print(out1.shape, out2.shape)               # torch.Size([1, 256, 64]) twice

Treat it as a research building block. There are no widely adopted pretrained checkpoints, and behaviour can differ from the papers because the original descriptions leave details open.

Comparison table

	Transformer	Mamba / SSM	Titans	MIRAS models (Moneta, Yaad, Memora)
Short-term memory	Full attention	Recurrent state	Sliding-window or segment attention	None (attention-free)
Long-term memory	KV cache, grows with length	Fixed-size state	MLP weights updated at test time	MLP weights updated at test time
Cost per token as context grows	Grows	Constant	Roughly constant plus chunked updates	Roughly constant plus chunked updates
Recall of far tokens	Excellent within window	Weak	Reported strong on BABILong and NIAH	Reported strong on NIAH
Learns at test time	Only in-context	No weight change	Yes	Yes
Tooling maturity	Very high	High	Community code	Research only
Largest size in first paper version	Not applicable	Not applicable	760M	1.3B

A decision list

Inputs fit a standard context window: use a Transformer.
You need long inputs and fast inference today: start with production-grade state-space or hybrid models, which have pretrained weights and serving support.
You research long-context memory: reproduce Titans or MIRAS from the community code, compare against Mamba and a sliding-window Transformer on your own data, and report chunk size.
You forecast drifting series: run ARIMA, a seasonal naive model and a pretrained forecaster first; try neural memory only if a clear gap remains on long histories.
You build tabular models on large or streaming tables: keep in-context learning as the core, prototype memory for overflow rows, and test that row order does not change predictions.
You need auditability: be cautious; weights that change during inference are harder to reproduce and explain.

Frequently asked questions

Is "Transformer 2.0" an official Google name?

No. Google uses the names Titans, MIRAS, ATLAS, Nested Learning and HOPE. The phrase comes from commentators, including a YouTube video by Matthew Berman about Titans and a January 2026 blog headline about Nested Learning. It is a search term, not a model name.

Does Titans replace attention?

No. Titans keeps attention for precise short-term context and adds a neural long-term memory beside it, as extra context (MAC), through a gate (MAG) or as a layer (MAL). The MIRAS models Moneta, Yaad and Memora, by contrast, are attention-free.

What does "learning at test time" mean here?

Part of the model, the memory MLP, changes its weights while processing input, by taking gradient steps on an associative memory loss. The rest stays fixed after training. Standard Transformers change no weights during inference; their in-context learning lives entirely in activations.

Did Titans really handle 2 million tokens?

The paper states that Titans scales beyond a 2-million-token context with higher needle-in-haystack accuracy than baselines, and Google's blog repeats it. But the first-version models are at most 760M parameters, and an independent reimplementation found the gains uneven. Treat it as a promising research result, not a settled property.

How is this different from fine-tuning?

Fine-tuning changes the main model's weights in a separate training job with labelled data. Titans memory updates a small inner network continuously, chunk by chunk, during inference and without labels, and its forgetting gate discards what is no longer useful. It is closer to a fast scratchpad than to retraining.

Summary

Titans, MIRAS, ATLAS and Nested Learning are one Google Research line that gives sequence models a long-term memory made of a small neural network, updated at test time by a surprise signal with momentum and forgetting. MIRAS turns the recipe into four design choices and yields the attention-free Moneta, Yaad and Memora; Nested Learning extends it to memories updating at many speeds, with HOPE as a proof of concept. The reported results are promising but come from models of 1.3 billion parameters or fewer, without official code, and an independent reimplementation found the gains uneven. It suits long, drifting time series and could extend tabular in-context learning past its row limit, but both are outlooks to test against simple baselines. "Transformer 2.0" is a nickname; the substance is a new kind of memory worth watching.

Transformer 2.0? Titans, MIRAS and memory that learns at test time