Good training pairs, not scale alone, bring AI models to a shared representation

Do AI systems trained on entirely different kinds of data, such as text and images, end up with the same internal picture of the world? A preprint by Randall Balestriero and Yann LeCun, posted to arXiv and not yet peer reviewed, argues that they can, and that the choice of training examples, not scale alone, governs how efficiently they get there. The authors report a mathematical proof that one family of models converges on a shared representation under specific conditions. They also describe experiments in which a 109-million-parameter text model lines up with image models better than every language model they tested, including models of up to 72 billion parameters.

The starting point is the Platonic Representation Hypothesis. This is the conjecture that models trained on different modalities (types of input, such as words or pixels) converge on the same representation as they scale up. A representation is the internal numerical encoding a model assigns to its input. Without a proof, the authors note, the hypothesis offers neither a guarantee nor a recipe. Their theorem concerns Joint-Embedding Predictive Architectures, or JEPAs, which learn by predicting the encoding of one input from the encoding of a related one. The two related inputs are called a positive pair. The theorem states that JEPAs trained separately on any number of modalities learn the same representation up to a single rotation, provided each positive pair consists of two observations of one scene. The authors call such pairs good views. Being the same up to a rotation means the models' internal spaces match apart from their orientation, much as two accurate maps of one city might be drawn with north pointing in different directions. For other kinds of pairs, the theorem bounds how far the representations can diverge.

The authors then argue that next-token prediction, the standard way of training large language models by having them guess the next piece of text, is itself a JEPA with bad views. Its pairs are parts of a single document rather than two observations of one scene. On their account, this explains why language models align with vision models only through scale. They report that experiments ranging from 145,000 captions up to the texts of 940 million web images support the theory. The experiments also reveal a scaling law of alignment that predicts the outcome of unseen training runs to within seed noise, meaning the run-to-run variation caused by random settings alone. On the same corpus, good views needed 42 times fewer training samples than next-token prediction to reach the same alignment. Their 109-million-parameter text JEPA aligned with the image encoders of DINOv2 and CLIP, two widely used vision models, better than every language model evaluated. It also beat CLIP's own text encoder.

The authors offer two demonstrations. In the first, a single rotation turned separately trained encoders into an image captioner. It scored 0.838 on CIDEr, a standard measure of how closely machine-written captions match human ones, on the COCO image collection. In the second, the authors report that finding the rotation does not even require matched examples. On what they describe as the latest pair-free benchmark, their text JEPA beat every zero-shot accuracy reported for language models of up to 8 billion parameters, where zero-shot means performing a task without being trained on it. It reached 64 per cent top-5 accuracy on ImageNet-100, a 100-category image classification test, compared with at most 56 per cent for those models. Top-5 accuracy counts an answer as correct when the right label appears among the model's five best guesses.

The authors conclude that positive views, and not scale alone, are the lever for building such shared representations. Their guarantee, however, depends on good views being available. For other kinds of pairs, the theorem limits the misalignment rather than removing it. The comparisons cover only the language models the authors evaluated and the accuracies already reported on the benchmark, and the findings have yet to pass peer review.