V*Bench: A Visual QA Benchmark with Detailed High-resolution Images

V*Bench is a visual question & answer benchmark that evaluates MLLMs in their ability to process high-resolution and visually crowded images to find and focus on small details.

Overview

**V*Bench** is a visual question & answer benchmark that evaluates MLLMs in their ability to process high-resolution and visually crowded images to find and focus on small details. This benchmark is in contrast with common multimodal benchmarks that focus on large, prominent visual elements.

V*Bench was introduced in V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs. Penghao Wu, Saining Xie, 2023.

This implementation evaluates performance of MLLMs on the benchmark and does not implement V* visual search algorithm introduced in the paper.

Dataset and Tasks

The dataset (craigwu/vstar_bench) consist of 191 multiple choice questions in two categories:

vstar_bench_ar - Attribute Recognition task has 115 samples and requires the model to recognize a certain type of attribute (e.g. color, material) of an object. This task has 4 multiple choice answers.

vstar_bench_srr - Spatial Relationship Reasoning task has 76 samples and asks the model to determine the relative spatial relationship between two objects. This task has two multiple choice answers.

Evaluation

First, install the inspect_evals Python package with:

pip install git+https://github.com/UKGovernmentBEIS/inspect_evals

Then, evaluate Attribute Recognition and Spatial Relationship Reasoning tasks:

inspect eval inspect_evals/vstar_bench_ar --model openai/gpt-4o
inspect eval inspect_evals/vstar_bench_srr --model openai/gpt-4o