ASIA
    ←Latent
    Dispatch · Video AI evaluation · 3 min
    SEOUL · Sejong University

    Over half the questions in 14 video AI benchmarks can be answered without real viewing

    Researchers at Sejong University and NAVER Cloud pared 24,416 video test questions to 11,033 that survived shortcut checks; on those, even the top-scoring model got fewer than half right.

    22 November 2026 · arXiv preprint, not yet peer reviewed
    Photo: Umut Tuluoglu

    Much of what video-understanding tests reward may have little to do with watching video. In a 2026 preprint, not yet peer reviewed, researchers at Sejong University and NAVER Cloud in South Korea audited 14 benchmarks, the standard question sets used to score video-language models: AI systems that take in video and answer questions about it in text. Of 24,416 multiple-choice questions, on clips lasting from seconds to hours, only 11,033 survived, roughly 45%. Most of the rest could be answered without genuinely using the pictures or the order of events. On the survivors, openly released models scored between 26.3% and 36.8%, against 25.6% for random guessing. That matters because benchmark scores are the usual yardstick for claims that models can follow video at all.

    The authors' concern is that a high score cannot separate perception from reasoning or from language priors, the background knowledge about how things usually go that a model absorbs from text. Many benchmarks build questions around a few key frames, or have a language model write them from a transcript, so it is unclear whether they test what is specific to video: continuity over time, cause and effect, a story told across scenes. The authors report that benchmarks with more shortcut questions tend to post higher accuracy. Instead of building another benchmark, they propose a shared way to vet existing ones.

    Their audit, Video-Oasis, applies six handicaps. Three remove the picture: models see only the question and options, a transcript of the soundtrack, or a string of captions describing the video at fixed intervals. Three remove time: models get only the middle frame, the frames in shuffled order, or an image-text matcher in the style of CLIP, which scores how well each frame fits the question with no sense of sequence. Across all 14 benchmarks, the models stayed above chance under every handicap: 33.5% to 36.2% with no video at all, and above 50% with shuffled frames. Each test uses three models, and a question counts as a shortcut only if all three get it right; had a single correct model sufficed, 92.7% of questions on average would have qualified.

    Further checks sent doubtful items to human reviewers, including shuffled-frame cases that might in fact depend on order; of 1,758 such questions, 804 were refined after review. To validate the flags, the team used three systems that took no part in filtering. Under normal conditions these answered on average about 76% of flagged questions correctly, against 28.6% to 39.5% on the survivors. The surviving set overlaps only 44.6% with the 6,609 questions that three of the diagnostic models all got wrong, which the authors read as evidence that it is not merely a pile of hard items.

    On the surviving questions, Gemini-2.5-Pro led with 46.7% and topped all five categories, which run from fine perceptual detail to global narrative; GPT-4o managed 27.5%. Agentic systems, which break a task into steps and reason over it in loops, diverged even with GPT-5-mini doing the reasoning in both: STAR reached 39.5%, VideoTree 30.1%. Global narrative, stitching together many scenes across a long video, was the weakest category for nine of 13 models, and eight of the nine open models scored below 25% there. In a test on 2,945 questions from two benchmarks, handing Eagle2.5 the correct time span, an oracle unavailable in practice, lifted its accuracy on surviving questions from 35.0% to 50.8%, but on shortcut questions only from 78.0% to 80.8%. The authors' comparisons of training methods pit existing models built on the same base but trained in different ways, so they are best read as early hints.

    Open questions remain. The authors acknowledge that captions or shuffled frames may leak timing cues, that a correct answer under a handicap does not prove a shortcut, and that the six tests overlap. The 55% figure also depends on the choice of diagnostic models and the all-three threshold. Some diagnostic models are themselves scored on the filtered set, which may pull their results down. And since the independent models missed about a quarter of flagged questions, not everything discarded was easy.

    Paper
    Video-Oasis: Rethinking Evaluation of Video Understanding
    Authors
    Geuntaek Lim, Sungjune Park, Jaeyun Lee, Inwoong Lee, Taeoh Kim, Dongyoon Wee, Minho Shim, Yukyung Choi et al.
    Venue
    arXiv preprint — not yet peer reviewed · 2 July 2026
    Link
    arxiv.org/abs/2603.29616
    Index
    stacks.braintank.ai
    Code
    github.com/sejong-rcv/Video-Oasis
    Cite

    Keep reading

    Photo: Umut Tuluoglu

    Latent · multimodal training data

    Training AI on questions that link several image regions lifts it on 25 benchmarks

    ZwZ2 learns from 20.4K automatically checked questions built to require combining several parts of one picture, and its gains extend well beyond fine-grained perception.

    arXiv preprint · 27 October 2026
    Google DeepMind 官方 Visualising AI 系列
    Google DeepMind 官方 Visualising AI 系列

    Photo: Google Deepmind

    Latent · Speculative decoding

    Small drafter speeds up diffusion text models by guessing which blank they fill next

    A 0.62M-parameter helper predicts the order in which a diffusion language model commits tokens, cutting the big model's passes without changing its output.

    arXiv preprint · 26 October 2026

    Photo: Brecht Corbeel

    Latent · Video stylisation

    Video model restyles one object or its background while largely sparing the rest

    Xiaomi's LociArt pairs a mask that encodes both where and which style with 100,000 synthetic training pairs, though a rival fine-tuned on the same data comes close.

    arXiv preprint · 27 October 2026