Self-training AI decides for itself when its practice tasks have gone stale

Teaching an AI model to improve itself raises a question that is easy to overlook: how does it know when the exercises it is practising on have stopped teaching it anything? Researchers at Xiaomi Research propose an answer in a preprint posted to arXiv, which has not yet been peer reviewed. Their method, A-RSI, watches the model's own training logs for signs that learning is tailing off and then brings in a fresh batch of tasks. On a suite of eleven benchmarks, the authors report, the approach raised average performance from 57.77 to 61.05 using 800 new tasks, with no external training examples and no human-written answers.

The work belongs to a line of research known as recursive self-improvement, in which a model produces its own training material, learns from it and repeats the cycle. According to the authors, existing methods mostly concentrate on what experience to generate next. They give less thought to when fresh experience should be introduced. Many also build their training targets through self-consistency or majority voting. In these methods the model answers the same question several times, and its most common answer is treated as the correct one. The drawback, the authors argue, is that supervision then rests on the model's own uncertain predictions.

A-RSI tackles the two issues separately. To decide when to change tasks, it monitors reward variation and accuracy recorded during training. The reward is the score the training process gives each attempt. The authors use these measures to judge when the current task pool, the set of problems the model is working through, has lost its learning signal, and at that point the pool is refreshed. The decision is made without consulting validation or test scores. Those are the held-out measurements researchers normally use to judge a model, so training is not steered by the same numbers used to evaluate it.

To make sure the new tasks have trustworthy answers, the system draws on a static library of programs written by a model. These programs generate multimodal questions, meaning questions that involve more than one kind of input, such as images alongside text. The programs also work out the reference answers. Because each answer comes from a program rather than from a vote among the model's guesses, it can be checked independently. The authors say they wrote down a specification for how tasks are produced, which they call a task-generation contract. They also automatically verified chosen properties of the reward and refresh rules, and audited the generated answers in a separate step.

Several things remain unclear from the abstract alone. It does not name the model that was trained, break down the gain of roughly 3.3 points across the eleven benchmarks, or say what the audit of generated answers turned up. The figures are the authors' own, measured on what they describe as their primary benchmark suite, and they have yet to face peer review.