ConViS-Bench:
Estimating Video Similarity Through Semantic Concepts

Benedetta Liberatori 1, Alessandro Conti 1, Lorenzo Vaquero 2, Yiming Wang 2, Elisa Ricci 1,2, Paolo Rota 1

1 University of Trento
2 Fondazione Bruno Kessler

What does it mean for two videos to be similar? Videos may appear similar when judged by the actions they depict, yet entirely different if evaluated based on the locations where they were filmed. While humans naturally compare videos by taking different aspects into account, this ability has not been thoroughly studied and presents a challenge for models that often depend on broad global similarity scores. Large Multimodal Models (LMMs) with video understanding capabilities open new opportunities for leveraging natural language in comparative video tasks. We introduce Concept-based Video Similarity estimation (ConViS), a novel task that compares pairs of videos by computing interpretable similarity scores across a predefined set of key semantic concepts. ConViS allows for human-like reasoning about video similarity and enables new applications such as concept-conditioned video retrieval. To support this task, we also introduce ConViS-Bench, a new benchmark comprising carefully annotated video pairs spanning multiple domains. Each pair comes with concept-level similarity scores and textual descriptions of both differences and similarities. Additionally, we benchmark several state-of-the-art models on ConViS, providing insights into their alignment with human judgments. Our results reveal significant performance differences on ConViS, indicating that some concepts present greater challenges for estimating video similarity. We believe that ConViS-Bench will serve as a valuable resource for advancing research in language-driven video understanding.

Paper    Code    Dataset

NeurIPS 2025 / Dataset & Benchmarks Track

Concept-based Video Similarity estimation

We introduce Concept-based Video Similarity estimation (ConViS), a task that quantifies video similarity along specific semantic concepts (e.g., location), and ConViS-Bench , a dataset of video pairs annotated with concept-level similarity scores (1-to-5) and free-form descriptions of similarities and differences. This bridges the gap between prior work focused solely on global similarity or differences in natural language

Video Pair Samples

ConViS-Bench contains 610 video pairs, each annotated with human-judged similarity scores across five general-purpose concepts. In addition to quantitative similarity scores, each pair is accompanied by free-form descriptions highlighting shared and differing elements, offering qualitative insight into human reasoning. It covers the broadest range of domains to date, i.e., 16 in total and features longer videos on average.

Main Action: 4.0
Main Subjects: 3.1
Main Objects: 2.8
Location: 2.4
Order of Actions: 3.9
workout, footwork, training, exercise, knee tucks, main action, green floor, same workout, syntetic grass floor place, gender, fitball, setting, location, gym tool, equipment, equipments, prone tucks, floor tucks, gliding discs, gender of trainers
Main Action: 4.3
Main Subjects: 2.7
Main Objects: 4.0
Location: 4.5
Order of Actions: 4.2
track, sport, running, athletics, relay race, before race, running track, track and field, introduce runners, presentation of athlethes men, race, women, gender, camera close-up, number of meters, genre of subjects, audience numerosity, sex of participates
Main Action: 4.0
Main Subjects: 4.1
Main Objects: 3.6
Location: 2.1
Order of Actions: 3.6
wool, sewing, crochet, needles, knitting, stitching, thread work, crochet hooks, point of view pink, wool, blue, color, speed, indoor, colors, outdoor, location, wool color, hand speed, ball of yarn, red and green, colour of thread
Main Action: 4.5
Main Subjects: 1.5
Main Objects: 4.0
Location: 4.0
Order of Actions: 2.5
action, training speed, action object, first one is performed on a wig, second is a complete hair cut (with coloring)
Main Action: 2.0
Main Subjects: 5.0
Main Objects: 1.5
Location: 5.0
Order of Actions: 1.5
chief, cooking, location, same person, cutting board, outdoor kitchen corn, food, order, manner, skillet, ingredients

Citation

If you find our work useful in your research, please consider citing: 


@inproceedings{liberatori2025convisbench,
title={ConViS-Bench: Estimating Video Similarity  Through Semantic Concepts},
author={Benedetta Liberatori and Alessandro Conti and Lorenzo Vaquero and Yiming Wang and Elisa Ricci and Paolo Rota},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2025},
url={https://openreview.net/forum?id=NoIWLerNKH}
}