How vision-language models are being turned into robot controllers, either as end-to-end vision-language-action policies or as agents driving a robot through an interface, and how well each approach holds up in the physical world.
Black wheeled robot with mecanum wheels in a white ribbed hall · Photo: Kinsey Wang
A 2026 preprint gives a frozen vision-language model a small command vocabulary and at most one example. In simulation it rivals robots trained on task data.