Özer Cem KELAHMETKELAHMET
Articles · Trends

Memorilla packs a whole document store into 16 memory tokens for a frozen LLM

A Stanford-led team has released Memorilla, a 194M-parameter module that reads a whole store of document embeddings and hands a frozen Qwen3-8B just 16 memory tokens. Here is how latent memory works, how it differs from RAG and KV-cache compression, and what it costs.

At a glance
Reading time
16 min
Sections
7
Code examples
1
Tables
1
Sources
11
Published
16 Nov 2026

Most language model memory systems work by pasting text into the prompt. A retriever finds a few chunks, the application glues them in front of the question, and the model reads them as ordinary tokens. That works, but it is expensive. Every chunk costs tokens, every token costs compute, and long histories quickly fill the context window.

Memorilla takes a different route. It is a small trainable module that sits between a document store and a frozen language model. Instead of passing text, it passes 16 vectors that live in the model's input-embedding space. The authors, led by researchers at Stanford, report that this setup beats strong baselines on five of eight benchmarks while using far fewer tokens to represent the source documents. The paper appeared on arXiv in October, checkpoints and data are on Hugging Face under an Apache-2.0 licence, and the release drew wide coverage as an open-source launch around 14 November.

This matters because it is a clear, public example of latent memory: memory stored and delivered as learned vectors rather than text. This article explains how it works, how it compares with retrieval-augmented generation (RAG) and with KV-cache compression, and when it is worth the trouble. If you want test-time memory inside the model itself, see our earlier piece on Titans and MIRAS. Memorilla is a different idea: the model stays frozen and memory arrives from outside.

What happened

Stanford-led team posts the Memorilla paper (15 October 2026)

The paper "Memorilla: Latent Semantic Memory for LLMs" was submitted to arXiv on 15 October 2026 (arXiv:2610.21375). The authors are Adibvafa Fallahpour, Parsa Idehpour, Vignesh Kothapalli, Nikita Mounier, Shirley Wu, Shayan Pardis and Jure Leskovec. The paper lists affiliations at Stanford University, the University of Toronto, the Vector Institute, the University of Pennsylvania and MIT.

The abstract states the goal plainly: compress an entire document collection into a fixed number of query-dependent memory tokens for a frozen LLM. Documents are encoded once into reusable embeddings. A learned module then picks out what matters for each question and writes it into the model's input space.

The headline results, as reported by the authors:

  • With 16 memory tokens, Memorilla outperforms baselines including Mem0 and MemoRAG on five of eight benchmarks covering personalisation, long-document question answering and knowledge-intensive question answering.
  • On FactKG it reaches 84.33% accuracy, against 80.43% for full-context inference, while using 108 times fewer tokens to represent the source documents.
  • Across all benchmarks it averages a score of 49.68, against 41.92 for xRAG, 40.75 for LLMLingua-2 and 38.02 for Mem0. The authors say it uses 394 times fewer document tokens than full context on average.
  • In a TextWorld agent experiment, average success rose from 18.6% with full-history prompting to 24.2%.

The project page at memory.stanford.edu links the paper, a GitHub repository (snap-stanford/memorilla) and a Hugging Face organisation. The Hugging Face model card for Memorilla-Qwen3-8B lists an Apache-2.0 licence. It describes five checkpoint folders (stage1, stage2, stage3, pv4 and pmv2), each holding a memory.pt file of 194M parameters in bfloat16. The card says each checkpoint was trained with a frozen Qwen3-8B decoder and a frozen Qwen3-Embedding-4B encoder, and must be used with those two models. Two datasets sit alongside it: Memorilla-Data and PersonalizationV4.

One caution: when we checked the public GitHub landing page, it showed little beyond a README. The model card points to the GitHub code for loading and evaluation, so check the repository's current state before you plan work around it.

Coverage frames Memorilla as an open-source release (14 November 2026)

On 14 November 2026, tech news sites including Emergent covered Memorilla as an open-source latent memory module from Stanford researchers, repeating the 16-token and FactKG claims. All benchmark numbers come from the authors' own evaluation; we found no independent reproduction.

How latent memory works: from soft prompts to memory tokens

A language model does not read words. It reads vectors. The first layer turns each token into an embedding, a list of numbers (4,096 of them for Qwen3-8B, per the paper). After that, the model only sees vectors. Nothing forces those vectors to come from real words.

If you can produce vectors in the right space, you can feed the model information that never existed as text.

Soft prompts. Prompt tuning, described by Lester, Al-Rfou and Constant in 2021, learns a few vectors by backpropagation and places them in front of the input. The model stays frozen. Only those vectors change. The authors found this approach became competitive with full model tuning as models grew past billions of parameters, and that one frozen model can serve many tasks. The catch is that a soft prompt is fixed per task. It teaches the model how to behave, not what a specific document says. Our prompt tuning page covers the details.

Gist tokens. In 2023, Mu, Li and Goodman showed that a model can learn to squeeze a prompt into a few "gist" tokens and reuse them. They reported up to 26 times compression, up to 40% fewer FLOPs and a 4.2% wall-time speed-up, with minimal quality loss, on LLaMA-7B and FLAN-T5-XXL. Here the compression is learned during instruction fine-tuning of the model itself, so the model is not frozen.

xRAG projection. xRAG (Cheng and colleagues, 2024) took a step closer to Memorilla. A dense retriever already turns each document into one embedding. xRAG treats that embedding as a new "modality" and learns a bridge that maps it into the language model's input space. In the words of its abstract, "the only trainable component is the modality bridge, while both the retriever and the language model remain frozen." Each retrieved document becomes one token. The authors reported an average improvement of over 10% across six knowledge-intensive tasks and a 3.53 times cut in overall FLOPs.

Memorilla. Memorilla keeps xRAG's frozen encoder and frozen decoder but changes what sits between them. Instead of projecting one retrieved document at a time, it reads the whole store and writes a fixed budget of tokens.

An everyday analogy helps. Plain RAG is like handing a colleague a stack of photocopied pages. xRAG is like handing them one index card per page. Memorilla is like asking a research assistant to read the whole filing cabinet with your question in mind and hand over a single page of notes, always the same length, written in a shorthand only that colleague understands.

Inside Memorilla: the four steps

The paper describes the pipeline in four stages.

1. Encode documents once. A frozen Qwen3-Embedding-4B model turns each document chunk into one vector of 2,560 numbers. What counts as a chunk depends on the task: one conversation session, one tweet, one knowledge-graph triple, or a passage of about 128 or 256 tokens. These vectors form a persistent store. See our embeddings page for background.

2. Condition the slots on the question. Memorilla keeps K = 16 learned "slot" vectors. They are shared across all questions and collections. For each query, the slots attend to the query's embedding, so they start out tuned to what is being asked.

3. Read the whole store. The conditioned slots then run through cross-attention blocks over every stored embedding. The configuration table lists two layers with eight attention heads. The paper describes this as a Perceiver-style latent-array design: a small fixed set of latent vectors reads a large input of any size. If that pattern is new to you, our Perceiver IO page explains it. The important point is that this is a soft, continuous read. There is no top-k cut-off where a relevant chunk can fall just outside the list.

4. Project into the LLM's input space. A linear projection maps each slot from 2,560 to 4,096 dimensions, the input size of Qwen3-8B. The 16 resulting vectors are placed in front of the question's normal token embeddings. The decoder is unchanged.

Training. Only the memory module and its projection learn. The loss is the negative log-likelihood of the answer tokens, with no loss on the prompt or memory positions. Training runs in three supervised stages:

  1. About 1.98 million English Wikipedia paraphrase-reconstruction examples, which teach the slots to carry content.
  2. About 1.34 million examples from ten datasets covering reading comprehension, open-domain question answering and summarisation.
  3. Fine-tuning on a mix of the eight benchmarks' training splits, with extra specialisation for two personalisation benchmarks.

The supervised stages ran in bf16 on four NVIDIA H100 GPUs, per the paper. The trainable module has 194,123,776 parameters in the 8B setup, about 2.4% of the frozen decoder's size.

For the agent experiment, the team used a frozen Qwen3-4B-Instruct-2507 with eight memory tokens and trained the module with GRPO, a reinforcement learning method, on the game's reward.

Latent memory vs RAG vs KV-cache compression

Memorilla competes with three common ways of giving a model more knowledge at answer time. They work at different layers of the stack, so they are not perfect substitutes.

Table 1. Latent memory vs RAG vs KV-cache compression
Approach What reaches the model What is trained Can a human inspect it? Main cost
RAG with raw text chunks Top-k chunks as normal tokens Usually nothing (retriever optional) Yes Prompt tokens grow with k and chunk size
Prompt compression (LLMLingua-2) Shortened text A small token classifier Yes, mostly Some detail is dropped as text
Soft prompts / gist tokens A few learned vectors Prompt vectors, or the model itself (gist) No Fixed per task, or requires tuning the model
xRAG One projected vector per retrieved document A projection bridge No Still depends on top-k retrieval
KV-cache compression (SnapKV) Fewer cached keys and values inside the model Nothing No Full prompt must still be read once
Memorilla A fixed 16 vectors from the whole store A 194M memory module per decoder No Per-model training; loses fine detail

Against RAG with raw text. Plain RAG, covered in our guide from naive to agentic RAG, keeps everything readable and model-agnostic. You can swap the LLM tomorrow and the index still works. Its weakness is that the prompt grows with the evidence, and a hard top-k cut can drop the chunk you needed. In the paper, a cosine-similarity RAG baseline with 50 chunks (RAG-50) scored 78.4% on FactKG against Memorilla's 84.33%. The authors also report that Memorilla uses 113 times fewer document tokens than RAG-50 on average.

Against text compression. LLMLingua-2 frames compression as token classification: a bidirectional encoder decides which words to keep. Its authors reported 1.6 to 2.9 times end-to-end speed-ups at 2 to 5 times compression. That is modest compression, but the output is still text, so any model can read it. LLMLingua-2 was the strongest method on PubMedQA in the Memorilla paper, at 69.00% against Memorilla's 59.00%.

Against KV-cache compression. The KV cache stores the keys and values the model computes for each token so it does not recompute them while generating. Methods like SnapKV prune that cache. SnapKV's authors reported 3.6 times faster generation and 8.2 times better memory efficiency at 16,000 input tokens, and up to 380,000 tokens on a single A100-80GB. But KV compression works after the model has read the prompt. You still pay to process the full text once per new context, and the cache is tied to one model. Memorilla moves that cost earlier: documents are embedded once by a smaller encoder, and the big model never sees them as tokens. The two ideas can combine. Prompt caching is a third neighbour: it avoids recomputing a shared prefix but does not shrink it.

Latency. The authors measured end-to-end latency with 512 documents on one H100 at batch size 1: 0.74 seconds for Memorilla against 19.9 seconds for full-context generation, about 27 times lower. They also note that the gains show mainly on large collections, because decoding time dominates when the store is small.

What it means in practice

Latent memory is not a drop-in replacement for RAG.

Where it helps. The strongest results came on personalisation and fact lookup across many small items. On PersonaMem-v2, Memorilla scored 44.60% against 30.90% for the best baseline (xRAG) and 25.64% with no context at all. On TriviaQA it reached 57.00 exact match, against 52.60 for full context. These are tasks where the answer depends on combining weak signals spread across many documents: a user's past chats, profile items or graph triples. A fixed-size read over the whole store suits that well.

Where it struggles. On NarrativeQA (questions about long stories) and PersonalizationV4, MemoRAG did better. On PubMedQA, Memorilla fell well behind several baselines. The authors say the method can lose fine-grained details such as exact names, probably because the encoder was not trained for compression. Sixteen slots are a hard budget. If the answer is a specific dosage, an invoice number or a quoted clause, a summary in shorthand is the wrong tool.

For agents. The TextWorld result points at continual learning. An agent's "documents" are its own past turns, and the store grows over time. The paper reports that full-history prompting ran out of its token budget on 71% of rollouts, against 58% for Memorilla, and that the agent revisited explored rooms less. Teams building agent memory should note that an untrained memory module scored below plain in-context learning, so training is not optional.

The real cost. The module is tied to one encoder and one decoder. Change either, and you retrain. It also needs open weights, because the module writes directly into the model's input-embedding space. Hosted APIs that accept only text cannot use it. Finally, the memory is opaque. You cannot show a user which passage the answer came from, which complicates audits and citations.

What to do now

  1. Keep RAG as your baseline. Measure any latent memory system against a well-tuned RAG setup before switching.
  2. Map your questions to the profile. List whether your answers depend on many weak signals (preferences, history, aggregated facts) or on exact strings (numbers, names, clauses). Latent memory suits the first; text retrieval suits the second.
  3. Reproduce one benchmark first. Use the released Qwen3-8B checkpoints and the evaluation scripts referenced on the model card to reproduce one published number on your hardware before trusting it.
  4. Test on your own data. The checkpoints were fine-tuned on the paper's benchmark splits. Expect to retrain stage 3 on your domain, and budget GPU time for it.
  5. Plan for model lock-in. Write down which decoder and encoder you would commit to, and what retraining costs if you upgrade. Our adapters page covers similar trade-offs for other frozen-model add-ons.
  6. Keep a citation path. If users or auditors need sources, run a parallel text retrieval to show evidence, or reserve latent memory for low-stakes personalisation.
  7. Measure latency at your real store size. The 27 times gain was at 512 documents. With a small store, decoding dominates and the gain may vanish.
  8. Fold it into context design. Treat memory tokens as one more input budget alongside system prompts and retrieved text. Our context engineering page shows how to plan that budget.

Risks and open questions

Fine detail can disappear. The authors state that Memorilla can lose exact names and other specifics. Sixteen vectors cannot hold everything, and the model may answer confidently from a blurred memory.

Results come from one model family. Only Qwen encoder and decoder pairs were tested. Transfer to other families, to code, and to images is untested by the authors' own account.

Opaque memory is hard to audit. Text retrieval leaves a trail. Latent tokens do not. That makes it harder to explain answers, remove a specific user's data from what the model "remembers", or detect a poisoned document in the store.

Evaluation is self-reported. All numbers so far come from the authors. Some baselines (xRAG and ComMer) used reported rather than re-run results where needed, which makes exact comparisons less clean.

The open-source package is still settling. The Apache-2.0 checkpoints are on Hugging Face, but the public code repository looked sparse when we checked. Confirm the training and evaluation code is complete before committing.

Watch next: independent reproductions on non-Qwen models, a release of full training code, results on code and multimodal data, and whether hosted model providers expose an input-embedding interface that would let modules like this work with closed models.

Frequently asked questions

What is Memorilla in one sentence?

Memorilla is a trainable 194M-parameter module that reads a store of document embeddings and writes 16 memory tokens into a frozen language model's input. In the released setup the model is Qwen3-8B and the encoder is Qwen3-Embedding-4B. Only the module is trained; the two Qwen models stay frozen.

How is it different from RAG?

RAG retrieves the top few chunks and pastes them in as text. Memorilla reads every stored embedding with cross-attention and compresses what matters into a fixed set of vectors. That cuts prompt size sharply, but the result is not human-readable and only works with the model it was trained for.

Is it the same as Titans or other test-time memory?

No. Titans-style memory changes weights inside the model as it reads. Memorilla keeps the model frozen and supplies memory from outside through the input embeddings. Our Titans and MIRAS article covers the in-model approach.

Can I use it with a hosted API model?

Not as released. The module writes vectors directly into the model's input-embedding space, which hosted text-only APIs do not accept. It also needs to be trained for each specific decoder, so it requires open weights.

Does it replace KV-cache compression?

No, the two solve different problems. KV-cache compression shrinks what the model stores after it reads a long prompt. Memorilla avoids putting the documents in the prompt at all, so the techniques could be used together.

Summary

Memorilla, from a Stanford-led team, is a clear example of latent memory: a 194M-parameter module that reads a whole store of document embeddings and hands a frozen Qwen3-8B 16 learned tokens. It builds on soft prompts, gist tokens and xRAG's frozen-model projection, but replaces top-k retrieval with a Perceiver-style read over the entire store. The authors report wins on five of eight benchmarks, 394 times fewer document tokens than full context on average, and 27 times lower latency at 512 documents. The trade-offs are real: lost fine detail, opaque memory, per-model training and a need for open weights. Use it where answers depend on many weak signals, keep text RAG where exact evidence matters, and wait for independent reproductions before betting a product on it.

Sources

Continue learning

Cite this article

Kelahmet, Ö. C. (2026, November 16). Memorilla packs a whole document store into 16 memory tokens for a frozen LLM. Özer Cem KELAHMET. https://www.ozercemkelahmet.com/articles/memorilla-latent-memory-tokens

BibTeX
@online{kelahmet2026memorilla,
  author = {Kelahmet, {\"O}zer Cem},
  title  = {{Memorilla packs a whole document store into 16 memory tokens for a frozen LLM}},
  year   = {2026},
  month  = {nov},
  url    = {https://www.ozercemkelahmet.com/articles/memorilla-latent-memory-tokens}
}

How this article connects

11 pages link to or from it. Drag to turn, click a dot to open it.

Version historyfirst version
  1. Published

Factual corrections are listed here with what changed. Wording and formatting edits are not listed.

Last updated Nov 16, 2026
Type to search.
    ↑↓ navigate↵ open