Memory for AI agents: how assistants carry knowledge between sessions

What it is

Memory for AI agents is the set of mechanisms that let an AI system keep information from earlier interactions and bring it back when it becomes relevant later. It is what allows an agent, a language model that plans and takes actions over many steps, to recall a user's preferences, the outcome of a previous task or a fact learned last week.

The underlying model does not change as it is used. Its weights are fixed after training, and each request starts afresh, so memory has to be built around the model. A typical architecture has three parts. In the write step, the agent or a separate process decides what is worth saving. The storage layer is often a database of text snippets indexed by numerical embeddings, and sometimes a structured store of facts or a graph of linked entities. In the read step, relevant items are retrieved and placed into the model's input. Many systems also summarise old exchanges, merge duplicates, update facts that have changed and discard what seems unimportant.

Designers often borrow labels from psychology. Working memory is whatever sits in the model's input right now. Episodic memory stores records of particular events, such as a conversation. Semantic memory holds distilled facts, such as a user's time zone. Procedural memory captures how to do things, such as routines the agent has found useful.

How it differs from retrieval-augmented generation (simple retrieval over past text) and a model's context window

A model's context window is the amount of text it can consider in a single pass. Anything inside it is directly available, and anything outside it is invisible. Windows have grown large but remain finite. Longer inputs cost more to process, and models do not always make reliable use of details buried deep in a long input. Memory is one way of choosing which small part of a long history earns a place in that window.

Simple retrieval-augmented generation searches a collection of documents or past text for passages resembling the current question and inserts them into the prompt. The collection is usually treated as fixed raw material. Agent memory goes further. The agent writes to its own store as it works, and over time entries are meant to be rewritten, consolidated, corrected or forgotten. In principle that turns a pile of transcripts into something closer to an organised notebook. Whether the extra machinery actually helps is a live question.

Where it is stuck

A large-scale evaluation, described in a preprint not yet peer reviewed, reports that dedicated memory systems often underperform simple retrieval. It suggests that the way they reorganise an agent's history can hide evidence, a cost its authors call memory tax.[1] Summaries and extracted facts are compact, but if the original is discarded, any detail they omit is gone.

Inference is another weak point. Recalling what was said is one thing, and noticing what a person tends to do across many loosely related exchanges is another. A benchmark built on hundreds of synthetic sessions, also an unreviewed preprint, found the best agent memory method it tested answering only about a third of habit questions correctly.[2]

Several other problems are widely recognised but unsettled. Systems must decide what to store and what to forget, and stop memories from going stale or contradicting one another. They must also keep an early mistake from being recalled and reinforced, and handle the privacy questions raised by retaining personal details. Evaluation is itself difficult, because the value of memory often shows only over long periods of use.