ASIA
    ←Agency
    Explainer · Chain-of-thought analysis · 5 min
    SEOUL · Seoul National University

    Labelling an AI's maths reasoning with the moves teachers look for

    A Seoul National University team uses decades of maths-education research to tag each step of a model's working, then uses the same vocabulary to shape its training.

    22 November 2026 · arXiv preprint, not yet peer reviewed
    3D render of metallic spheres linked by glowing wires on deep blue
    Photo: Steve A Johnson

    When a language model tackles a competition maths problem, the usual scorecard asks one thing: did the final answer match? That reveals little about how the model got there. Modern reasoning models write out a chain of thought, a step-by-step working that comes before the answer. Two models can reach the same correct number by very different routes. The same model can also stumble on a lightly reworded problem without anyone being able to say what changed in its approach.

    A 2026 preprint on arXiv tries to make those routes visible. Jonghyun Song, Yohan Jo and colleagues at Seoul National University borrow a vocabulary that maths teachers have used for decades to describe how human students solve problems. They apply it to models' chains of thought, use it to probe two open questions about how models reason, and then fold it back into training.

    The idea

    In maths education, a heuristic is a purposeful problem-solving move rather than routine calculation. Examples include simplifying the problem, splitting it into cases, introducing an auxiliary object, or working backward from the goal. Landmarks in this tradition include George Pólya's work in 1945 and Alan Schoenfeld's in 1985. Researchers in this tradition code transcripts of students thinking aloud, marking which moves they make and when. The authors also note that Schoenfeld found, in 1979, that explicitly teaching heuristics helped students more than practice on the same problems with worked solutions.

    The authors argue that earlier ways of analysing a model's chain of thought are coarse. These approaches measure its length, count self-correcting words such as "wait", mark broad phases such as planning and verification, or map its structure as a graph or tree. They say little about which specific mathematical action the model is taking. A fixed, problem-independent list of heuristics would let researchers compare strategy use across problems, across models, and before and after extra training.

    The team points this tool at two questions. The first is whether models fail on altered problems because they reuse learned techniques. The second concerns reinforcement learning with verifiable rewards, or RLVR, a training method in which a model is rewarded only when its final answer is checked as correct. One hypothesis, called mode-seeking, holds that RLVR mostly concentrates on strategies the base model already uses rather than teaching it new ones.

    How this paper puts it to work

    The framework, called SHAPE, defines eleven heuristic families drawn from the maths-education literature. They include changing representation, simplification, case analysis, contradiction, analogy, pattern exploration, working backward and verification. A twelfth catch-all category covers routine computation, repetition, unrelated statements and final answers. Each chain of thought is cut into content units, the largest continuous stretches that serve a single problem-solving purpose, and each unit gets one or more labels. A whole solution is then summarised as a heuristic-frequency distribution, something like a pie chart showing what share of its labels falls in each family.

    The labelling is done by prompting another language model. To pick that annotator, four of the authors, one of them a graduate researcher in maths education, agreed by consensus on labels for 48 solution traces. Those traces contained 1,598 content units. Of the candidates tested, Grok-4.1-Fast matched these human labels best, with an F1 score (a measure balancing wrongly added and missed labels) of 77.16%. The open-weight Qwen3.5-27B scored 70.44% and is the authors' stated default.

    The probes use MATH-Perturb, a set of difficult problems, each paired with two variants. In the "simple" variant the original solution method still works. In the "hard" variant a different method is needed. For six models, the authors used Jensen–Shannon divergence, a score where zero means two distributions are identical, to measure how much heuristic use shifted between each original and its variants. For the mode-seeking question, they compared four base models with versions trained by reinforcement learning. The comparison used two overlap measures. Density counts how many of the base model's solution neighbourhoods each trained-model solution falls into, on average. Coverage shows what fraction of those neighbourhoods the trained model reaches at all.

    Finally, they changed a training recipe. GRPO is a reinforcement learning method that scores each sampled answer against the others drawn for the same problem. In a variant called Plan-GRPO, the training prompt asks the model to state goals and subgoals. The authors' version, HA-Plan-GRPO, also lists the eleven heuristic families and asks the model to pick a relevant one and justify it before each major step. Data, reward and compute budget were otherwise held the same.

    What it shows, and what it does not

    All six models were less accurate on hard variants than on simple ones. Their heuristic mix also shifted more between original and hard problems than between original and simple ones, consistently and at statistical significance. That significance is uncorrected for the number of tests, and some gaps are small, such as 0.043 against 0.066. Because hard variants are built to need a different approach, some shift is expected. The measure shows that strategy use changed, not whether the change was the right one. The authors concede that inappropriate reuse may still occur.

    On mode-seeking, the trained models' successful solutions overlapped heavily with their own base models. Density ranged from 1.219 to 1.525 across the four pairs. Pairs of unrelated base models overlapped far less, with Density of 0.525 and 0.594 and Coverage of 0.348 and 0.437. Only the Olmo pair showed clearly reduced Coverage, at 0.707, which is the clearest sign of narrowing. The other three pairs ranged from 0.871 to 0.950, and one comparison baseline rests on few samples. The evidence fits the hypothesis but does not settle it.

    In training, HA-Plan-GRPO beat Plan-GRPO for all three small models tested, which ranged from 1.7 to 8 billion parameters. Mean average accuracy rose by 1.1 to 7.0 percentage points, and the chance of at least one correct answer in 16 attempts rose by 0.4 to 2.4 points. The authors' variant won 14 of 15 per-dataset comparisons on each measure. However, the largest gain partly reflects Plan-GRPO falling below the untrained model on that model, from 46.9 to 42.8, before HA-Plan-GRPO lifted it to 49.8. The results come from three training runs per variant without significance tests. Several gains fall within run-to-run variation. There is no comparison with plain GRPO, nor with an equally long prompt lacking the heuristics, so the benefit cannot yet be pinned on the heuristic vocabulary itself.

    The authors add their own caveats. SHAPE describes what a chain of thought says, not whether it faithfully reflects the model's internal computation. Borrowing a vocabulary built for human think-aloud transcripts does not make the two equivalent. The method handles single solution traces, not dialogue or teams of agents. Beyond those, the work is confined to maths, with automatic labels that agree with humans most but not all of the time.

    Paper
    SHAPE of Chain-of-Thought in Math Reasoning
    Authors
    Jonghyun Song, Sangjun Song, Minjae Oh, Haesung Pyun, Sungsik Lee, Yohan Jo et al.
    Venue
    arXiv preprint — not yet peer reviewed · 28 June 2026
    Link
    arxiv.org/abs/2608.28600
    Index
    stacks.braintank.ai
    Code
    github.com/holi-lab/SHAPE-of-CoT
    Cite

    Keep reading

    Photo: Guerrillabuzz

    Agency · coding agent evaluation

    Coding agents meet most of a user's requests but rarely finish the whole job

    A benchmark built from real user sessions finds the best of ten models passes an average of 69.9% of checklist items per task yet fully completes only 26.4% of web-app edits.

    arXiv preprint · 26 October 2026

    Photo: Laurens Van Der Drift

    Agency · Agent reinforcement learning

    Reinforcement learning for AI agents that hands out credit turn by turn

    An IQuest preprint trains search and tool-using agents by scoring each complete turn first, then sharing that score among its tokens, and tops five rival methods.

    arXiv preprint · 26 October 2026

    Photo: Shane Rounce

    Agency · agent training data

    Fixing broken practice tasks before AI agents learn office work from them

    WorkFactory maps the information each work task needs and patches gaps by test-running it, then fine-tunes smaller Qwen models on a stronger model's demonstrations.

    arXiv preprint · 27 October 2026