June 2025 — Appendix to the MVP paper
A word-by-word excerpt from the Behind the Scenes appendix of A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs (TMLR 2025), which came out of my internship at FAIR. See also the Behind the Scenes of my whole PhD.
In this section we go beyond what usually goes into a paper and discuss how the paper came about, what did not work, or what motivated the authors - so in essence: all the things that are usually deemed too subjective or “unscientific”, yet would help other researchers, especially those joining the field, often much more than the polished narrative of the main paper.
Several of the authors who work on video modeling felt a growing frustration with existing benchmarks that often rewarded the wrong things. So the direction of the project was quickly set after a short period of brainstorming: Quantify in what ways existing benchmarks are broken, and then fix it. We then spent a few months staring at hundreds of examples from the MVBench datasets, scouting for glaring issues or shortcuts and manually annotating lots of data. First, we tested the simple baselines with respect to frequency, text-only or single-frame biases, and soon included the less often discussed video-only (remove question) and Simple Socratic LLM shortcuts. In between we had philosophical discussions about benchmark design (bottom-down vs. top-down) or what it means for a video task to be truly temporal: is it temporal if two-frames are needed, or if a single frame is needed but it has to be a key-frame (needle in a haystack), or ...? Regarding benchmark design, should we adopt other paper’s taxonomies or design or own? Should one collect all kinds of examples and ad-hoc define a taxonomy (bottom-up), or should one define a taxonomy, then systematically collect examples to fit the taxonomy (top-down)? From the beginning the idea of minimal video pairs generated excitement among us: Minimal visual pairs have led to much progress in the field of vision-and-language compositionality (e.g. Winoground), yet had not been explored much in the realm of moving images.
There were, and perhaps still are, plans to crowd-source human shortcut performance, i.e. how much better are humans at solving video tasks when given single frames? At scale this could also be used to filter out examples with more precision than our five-model ensemble approach. Doing human crowd-sourcing well is not trivial, it is time-intensive and requires dedication but it can lead to much stronger insights than relying purely on automatic metrics and black-box models.
After the exploration phase, we executed on the benchmark building: From the start we had identified several promising datasets to mine minimal video pairs and continuously added a new source roughly every week. Perception Test was the first to go through our curation pipeline and hence took the longest as we were still refining the pipeline steps. Language Table was very hard to do well with many edge cases in the entailment detection, and also with its scale of 440K video-caption examples (imagine looking for potential pairs of videos, i.e. 440,0002 combinations).
Video-QA is becoming an increasingly popular topic; it is a very exciting direction with enough dimensions for everyone to innovate on: long video benchmarking, intuitive physics, social common-sense/Theory-of-Mind/narratives, novel simulation engines, and so on.
Despite our best efforts studying shortcuts, we probably missed some shortcuts or issues in MVP. It is good to think two steps ahead what kind of shortcuts future more capable models could take. It is easier said than done, but in retrospect older video benchmarks from 2015-2020 might have at least been able to address single-frame biases, a priori, during benchmark design. Instead now the field took years to identify and clean up benchmarks.