Agent memory
Agent memory lets AI agents keep context within a task and across sessions. Learn short-term and long-term memory, retrieval, summarisation and privacy design.
Overview
Agent memory is everything an AI agent uses to remember: what happened earlier in a task, what it learned in past sessions, and what it knows about the people and systems it works with. A language model on its own forgets everything once a request ends. Memory is the engineering around the model that gives it continuity.
A detective's notebook is a good picture. During a case, the detective keeps notes on clues, interviews and open questions, because nobody can hold a whole investigation in their head. Between cases, they keep files on recurring people, places and methods that worked. When a new case starts, they do not reread every old file; they look up what is relevant. Agent memory works the same way: a working notebook for the current task, and searchable files for the long term.
What it is
Memory is usually divided by how long it lasts and what it holds.
| Type | Lasts | Holds | Typical implementation |
|---|---|---|---|
| Working memory | One model call | The current context window | Prompt, recent messages, tool results |
| Short-term memory | One task or session | Conversation history, scratchpad, intermediate results | Message history, summaries, state objects, checkpoints |
| Long-term semantic | Across sessions | Facts and preferences | Key-value stores, documents, profiles |
| Long-term episodic | Across sessions | Records of past tasks and outcomes | Logged trajectories, searchable with embeddings |
| Procedural | Across sessions | How to do things | Instructions, skills, playbooks, learned rules |
Memory has three basic operations: write (decide what to store), retrieve (find what is relevant now) and forget (remove, expire or overwrite what is stale or unwanted).
Where it came from
Early chat systems simply resent the whole conversation each time, which stopped working once conversations exceeded the context window. Developers added rolling summaries and retrieval from vector databases. Research in 2023 explored agents with memory streams that scored memories by recency, importance and relevance, and systems that managed memory like an operating system manages main memory and disk. Commercial assistants introduced user-level memory in 2024. By 2025 agent frameworks and SDKs shipped memory components, and coding agents used project instruction files and notes as durable memory. "Context engineering", the discipline of deciding what goes into a model's context at each step, grew largely out of memory design.
How it works
Managing the context window
The context window is the agent's working memory. As a task grows, old messages and tool results crowd out new ones. Common techniques:
- Truncation: drop the oldest content.
- Summarisation: replace old turns with a compact summary.
- Selective retention: keep key facts, decisions and open questions in a scratchpad; drop raw outputs.
- Offloading: write large results to files or stores and keep only references in context.
- Sub-agents: let a separate agent process a large input and return a short result.
Writing long-term memory
The agent, or a background process, decides what is worth keeping: stable user preferences, important facts, decisions, outcomes of past tasks. Good systems store memories as small, specific statements with a source and a date, rather than whole transcripts.
Retrieving
At the start of a task or step, the system retrieves relevant memories. Methods include exact lookup by key (user id, project), semantic search with embeddings, keyword search, and filters on recency or type. Retrieved memories are added to the context, ideally with labels showing where they came from.
Updating and forgetting
Facts change. A preference stated last year may be wrong now. Memory systems need update rules (newer statements replace older ones), expiry for temporary facts, and deletion on request. Users should be able to see, correct and delete what an agent remembers about them.
Key concepts and terms
| Term | Meaning |
|---|---|
| Context window | The text a model can read in one call. |
| Working memory | What is in the current context. |
| Short-term memory | State kept for one task or session. |
| Long-term memory | Information kept across sessions. |
| Episodic memory | Records of specific past events or tasks. |
| Semantic memory | General facts and preferences. |
| Procedural memory | Knowledge of how to perform tasks. |
| Summarisation | Compressing past context into a shorter form. |
| Checkpoint | A saved snapshot of agent state that allows resuming. |
| Context engineering | Deciding what enters the context at each step. |
A worked example
A project assistant helps a small engineering team over months.
- Within a task: while preparing a sprint report, it reads twenty tickets. Instead of keeping every ticket in context, it writes a short scratchpad: completed items, blocked items, risks. Raw ticket text is dropped after it is summarised.
- Across sessions: it stores a few long-term facts: the team reports in English, the release day is Thursday, the lead prefers tables for status updates. Each fact has a date and the conversation it came from.
- Retrieval: next week, when asked for the report, it loads those facts and the previous report, so the new one follows the same format and highlights changes.
- Updating: when the lead says releases move to Wednesday, the stored fact is replaced, not duplicated.
- Forgetting: when a team member asks the assistant to forget a personal detail they shared, the memory is deleted and no longer retrieved.
Strengths and limitations
Strengths
- Continuity. Users do not have to repeat context every time.
- Longer tasks. Agents can work beyond the size of one context window.
- Personalisation. Stored preferences make outputs fit the user.
- Learning from experience. Records of past outcomes improve future decisions.
Limitations
- Stale or wrong memories. Old facts can mislead if not updated.
- Retrieval errors. Irrelevant memories distract; missed memories cause repetition.
- Privacy risks. Stored personal data must be protected, visible and deletable.
- Memory poisoning. Injected content can plant false memories that affect later sessions.
- Over-familiarity. Using remembered details where they are not relevant feels intrusive.
How it compares with alternatives
| Approach | Use it when |
|---|---|
| Long context only | Tasks fit comfortably in the window and sessions are independent |
| Summarisation | Conversations are long but the gist is enough |
| Retrieval-based memory | Many facts accumulate and only some are relevant at a time |
| Structured state | Workflows with known fields, such as order status or form progress |
| Fine-tuning | Stable, general behaviour should change permanently, not per user |
Practical guidance and best practices
- Store less, but better. Keep small, specific, dated facts with sources, not full transcripts.
- Separate memory by scope. User, project and organisation memories should not leak into each other.
- Retrieve deliberately. Load memories that change the answer; skip ones that only decorate it.
- Handle updates and conflicts. Newer statements replace older ones; ask when unclear.
- Give users control. Let people view, correct and delete memories.
- Protect against poisoning. Do not write memories from untrusted content without review.
- Exclude sensitive data by default. Avoid storing health, financial identifiers and similar data unless required and permitted.
- Evaluate memory effects. Test whether memory improves task success and whether wrong memories cause failures.
Common mistakes
- Saving everything. Memory fills with noise and retrieval quality drops.
- No expiry or update rules. Old facts persist and contradict new ones.
- Mixing users' memories. A serious privacy failure.
- Writing memories from web pages or emails. An open door for poisoning.
Frequently asked questions
Do language models have memory?
Not on their own. A model only sees what is in its current context. Memory is built around the model by storing information and adding relevant parts back into the context.
What is the difference between short-term and long-term memory in agents?
Short-term memory holds the state of the current task or session, such as recent messages and intermediate results. Long-term memory keeps facts, preferences and past outcomes across sessions.
Is a vector database the same as agent memory?
A vector database is one way to store and search memories by meaning. Agent memory also includes what to store, when to update or forget, and how to present memories to the model.
How do agents handle tasks longer than their context window?
By summarising old steps, keeping key facts in a scratchpad, offloading large data to files or stores, and using sub-agents that return compact results.
How should agent memory handle privacy?
Store only what is needed, keep scopes separate, exclude sensitive categories by default, and let users see, correct and delete what is remembered.
Glossary
- Embedding: a numeric vector representing meaning.
- Memory poisoning: planting false information in an agent's memory.
- Scratchpad: a working area for intermediate notes.
- Sub-agent: a separate agent that handles a delegated part of a task.
- Vector database: a store that searches items by embedding similarity.
Summary
Agent memory gives stateless models continuity. Working memory is the context window; short-term memory carries a task or session; long-term memory keeps facts, experiences and procedures across sessions. The core operations are write, retrieve and forget. Store small, dated, scoped facts, retrieve only what changes the answer, update and expire stale entries, protect against poisoning and give users control over what is remembered.
Related
It belongs to Coding and computer agents in my stack, under Agentic AI. Others in the same group: