Dedicated memory for AI agents often underperforms simple retrieval, study reports

Giving AI agents a memory sounds like an obvious improvement, but a new study suggests it often comes at a price. Researchers at the University of Illinois at Urbana-Champaign report that many purpose-built memory systems for AI agents perform worse, in many cases, than a far simpler approach: searching the agent's raw history directly. The work, posted on arXiv, is a preprint that has not yet been peer reviewed. Its interest lies in the question it raises about a component that is increasingly treated as essential for agents that operate over long periods.

The paper concerns what are known as long-horizon agents: AI systems, typically built on large language models, that carry out tasks over many steps or sessions and accumulate a lengthy record of past interactions. There are broadly two ways to let such an agent draw on that record. One is retrieval-augmented generation, or RAG, in which stored text is searched for passages relevant to the current request and those passages are supplied to the language model along with the prompt. In its plainest form, referred to in the paper as naive RAG, the search runs over the history exactly as it was recorded. The other is a dedicated memory system, which processes the history in advance, for example by reorganising facts or distilling a user's preferences, and stores the result for later use.

To compare the two, the authors ran what they describe as a large-scale evaluation across multiple benchmarks and model backbones, the underlying language models on which agents are built. They report that many dedicated memory systems underperformed naive RAG in many cases, meaning that the effort of building memory had, in those cases, a negative net value. The abstract does not name the systems or benchmarks involved, nor does it give figures for the size of the gap.

Using RAG as a reference point, the authors then split the net value of memory into two parts. Memory benefit is the gain a memory system delivers over direct retrieval; memory tax is the loss. Both, they argue, stem from the same source: memory processing changes how raw information is organised and accessed. The authors call these changes double-edged deviations from the raw history. They are introduced deliberately to help with the task, but some of them may also remove or obscure evidence that the agent needs. Put simply, tidying up a record can discard or bury the detail on which a later answer depends.

Since memory benefit reflects the gains a design is intended to deliver, the researchers concentrate on the tax, which they describe as overlooked. According to the abstract, they examine how it arises both from the design of memory systems and from the way those systems are evaluated, and they extend the concept to further applications of memory.

The authors present memory tax chiefly as a new lens, arguing that the field should pay less attention to how a memory system performs on its own and more to its net value against a simple baseline. The abstract leaves open how large the tax is in particular settings and how memory systems might be redesigned to keep their benefits while paying less of it. The authors say they will release their code and results, which should allow others to test the claims.