Multimodal
Build MMBench_V11 response matrix from VLMEval/OpenVLMRecords
Response matrix
Each row is an AI model and each column an item, ordered so the strongest models and easiest items gather toward one corner. 251 subjects × 3,579 items, 97% of cells evaluated. The heatmap shows a representative 753 of 3,579 items — evenly sampled across difficulty — so each cell stays square and legible.
Scale: 1 = correct · 0 = incorrect
Sample items
A spread of items across the difficulty range, each shown with a few subjects' actual answers.
Where is the man? A: The building on the left of the man B: The building is behind the man C: The building is next to the man D: The building on the right of the man
How subjects answered
D
D
B
Which letter marks the territory controlled by the ancient Maya civilization? A: A B: D C: B D: C
How subjects answered
D
D. C
Answer: B The ancient Maya civilization was located in the region that is now known as Mesoamerica, which includes parts of present-day Mexico and Central America. On the provided map, the letter "D" is located in the southern part of Mexico and covers a region that extends into Central America, which aligns with the historical territory of the ancient Maya civilization. Therefore, the correct answer is B, which corresponds to the territory controlled by the ancient Maya civilization.
who is this person? A: Meghan Markle B: Kate Middleton C: Emma Watson D: J.K. Rowling
How subjects answered
D
D
The answer is A. Meghan Markle. BECAUSE:
Which mood does this image convey? A: Angry B: Sad C: Anxious D: Happy
How subjects answered
D
D. Happy
D. Happy
What's the function of the demonstrated object? A: Providing multiple electrical outlets. B: Circulating air and creating a cooling breeze. C: Providing electricity. D: Carrying documents.
How subjects answered
B
B. Circulating air and creating a cooling breeze.
A. Providing multiple electrical outlets.
What's the profession of the people in this picture? A: florist B: lawyer C: librarian D: financial analyst
How subjects answered
A
A
A
Subjects
251 subjects, ranked by mean response (accuracy) across this benchmark's items.
+ 215 more subjects evaluated.