Rewarding how AI searches, not just what it answers, yields sturdier search agents

Training an artificial-intelligence search agent by grading each step of its hunt for information, rather than only its final answer, produces agents that cope better with unfamiliar questions, according to a study published in Nature Communications. The authors report that the improvement came mainly from the agents learning to write better search queries. They also found that the components they trained to judge those steps could be reused with other AI models, including commercial ones.

Large language models (LLMs), the systems behind modern chatbots, are changing how web search works. Instead of returning a ranked list of documents, they increasingly compose an answer directly. They are also being deployed as autonomous "agents" that search iteratively. Such an agent sends a query to an external knowledge source, reads what comes back and decides what to look for next. The authors argue that the difficulty lies in the middle of that loop, because these systems find it hard to generate good intermediate steps, such as well-chosen queries.

Previous approaches to training search agents, the authors write, have mostly relied on outcome supervision, in which the agent is rewarded only when its final answer is correct. According to the study, this has two weaknesses. It invites reward hacking, in which a system finds ways to collect its reward without actually doing the task as intended. It also encourages agents to lean on parametric memory, the knowledge absorbed into a model's internal parameters during training, instead of genuinely searching. Both habits limit how well agents generalise to out-of-domain tasks, meaning questions unlike those they were trained on.

The researchers are affiliated with the US National Library of Medicine at the National Institutes of Health, the University of Illinois Urbana-Champaign and the University of Virginia. To tackle the problem, they built a framework called RAG-Gym. RAG stands for retrieval-augmented generation, a technique in which a language model draws on documents fetched from an outside source when producing its response. RAG-Gym moves supervision away from the final answer and onto the search process itself. Using it, the team systematically examined three things: how agents are designed, how their parameters are optimised during training and how their individual actions are evaluated. One capability stood out as critical. The authors call it reasoning reflection, which broadly means an agent's capacity to look back over its own reasoning.

Building on that finding, they developed Re²Search++, an agent trained with process supervision. The authors report substantial improvements on multi-hop information-seeking benchmarks. These are standard test sets of questions that can only be answered by chaining together several pieces of information, and the largest benefit appeared in out-of-domain settings. The gains, they say, were driven primarily by higher-quality search queries rather than by optimising answers alone. The team also trained search critics, which are models that assess the quality of search steps. These critics transferred across different language models, including proprietary ones.

The abstract does not give the size of the improvements, and the evaluation rests on benchmark question sets rather than everyday use. The authors conclude that supervising the search process yields more reliable and generalisable information-seeking agents. The abstract does not address how far that holds beyond the benchmarks tested.