Over half the questions in 14 video AI benchmarks can be answered without real viewing
Researchers at Sejong University and NAVER Cloud pared 24,416 video test questions to 11,033 that survived shortcut checks; on those, even the top-scoring model got fewer than half right.

Much of what video-understanding tests reward may have little to do with watching video. In a 2026 preprint, not yet peer reviewed, researchers at Sejong University and NAVER Cloud in South Korea audited 14 benchmarks, the standard question sets used to score video-language models: AI systems that take in video and answer questions about it in text. Of 24,416 multiple-choice questions, on clips lasting from seconds to hours, only 11,033 survived, roughly 45%. Most of the rest could be answered without genuinely using the pictures or the order of events. On the survivors, openly released models scored between 26.3% and 36.8%, against 25.6% for random guessing. That matters because benchmark scores are the usual yardstick for claims that models can follow video at all.
The authors' concern is that a high score cannot separate perception from reasoning or from language priors, the background knowledge about how things usually go that a model absorbs from text. Many benchmarks build questions around a few key frames, or have a language model write them from a transcript, so it is unclear whether they test what is specific to video: continuity over time, cause and effect, a story told across scenes. The authors report that benchmarks with more shortcut questions tend to post higher accuracy. Instead of building another benchmark, they propose a shared way to vet existing ones.
Their audit, Video-Oasis, applies six handicaps. Three remove the picture: models see only the question and options, a transcript of the soundtrack, or a string of captions describing the video at fixed intervals. Three remove time: models get only the middle frame, the frames in shuffled order, or an image-text matcher in the style of CLIP, which scores how well each frame fits the question with no sense of sequence. Across all 14 benchmarks, the models stayed above chance under every handicap: 33.5% to 36.2% with no video at all, and above 50% with shuffled frames. Each test uses three models, and a question counts as a shortcut only if all three get it right; had a single correct model sufficed, 92.7% of questions on average would have qualified.
Further checks sent doubtful items to human reviewers, including shuffled-frame cases that might in fact depend on order; of 1,758 such questions, 804 were refined after review. To validate the flags, the team used three systems that took no part in filtering. Under normal conditions these answered on average about 76% of flagged questions correctly, against 28.6% to 39.5% on the survivors. The surviving set overlaps only 44.6% with the 6,609 questions that three of the diagnostic models all got wrong, which the authors read as evidence that it is not merely a pile of hard items.
On the surviving questions, Gemini-2.5-Pro led with 46.7% and topped all five categories, which run from fine perceptual detail to global narrative; GPT-4o managed 27.5%. Agentic systems, which break a task into steps and reason over it in loops, diverged even with GPT-5-mini doing the reasoning in both: STAR reached 39.5%, VideoTree 30.1%. Global narrative, stitching together many scenes across a long video, was the weakest category for nine of 13 models, and eight of the nine open models scored below 25% there. In a test on 2,945 questions from two benchmarks, handing Eagle2.5 the correct time span, an oracle unavailable in practice, lifted its accuracy on surviving questions from 35.0% to 50.8%, but on shortcut questions only from 78.0% to 80.8%. The authors' comparisons of training methods pit existing models built on the same base but trained in different ways, so they are best read as early hints.
Open questions remain. The authors acknowledge that captions or shuffled frames may leak timing cues, that a correct answer under a handicap does not prove a shortcut, and that the six tests overlap. The 55% figure also depends on the choice of diagnostic models and the all-three threshold. Some diagnostic models are themselves scored on the filtered set, which may pull their results down. And since the independent models missed about a quarter of flagged questions, not everything discarded was easy.
- Paper
- Video-Oasis: Rethinking Evaluation of Video Understanding
- Authors
- Geuntaek Lim, Sungjune Park, Jaeyun Lee, Inwoong Lee, Taeoh Kim, Dongyoon Wee, Minho Shim, Yukyung Choi et al.
- Venue
- arXiv preprint — not yet peer reviewed · 2 July 2026
- Link
- arxiv.org/abs/2603.29616
- Index
- stacks.braintank.ai
- Code
- github.com/sejong-rcv/Video-Oasis
- Cite

