Multimodal
FineGRAIN T2I failure-mode benchmark: ~17 text-to-image models x 760 prompts, each prompt tagged with one of 27 fine-grained failure modes (counting, colour/shape/texture attribute binding, spatial relations, physics, text rendering, negation, perspective, ...) across 11 categories. Each generated image carries a human label for whether the prompt's failure mode is present; response is the human success verdict (1 = no failure / prompt rendered correctly, 0 = failure).
Response matrix
Each row is an AI model and each column an item, ordered so the strongest models and easiest items gather toward one corner. 17 subjects × 760 items, 99% of cells evaluated.
Fit to width. Hover for subject & item; click a cell for details.
Scale: 1 = correct · 0 = incorrect
Sample items
A spread of items across the difficulty range, each shown with a few subjects' actual answers.
An arrangement of exactly two red apples and precisely three yellow bananas on a circular plate. Blur background, product photography.
How subjects answered
images/qwen-image/counts_or_multiple_objects/0.png
images/gpt_image1/counts_or_multiple_objects/0.png
A cyclist speeding down a hill, with frames capturing the downhill descent. Highlight the motion of the legs pedaling and the wheels spinning rapidly.
How subjects answered
images/wan22/action_and_motion_representation/21.png
images/seeDream3/action_and_motion_representation/21.png
A fabric tree trunk with a velvet texture, standing in a forest of regular trees. The forest is lit by the soft glow of the sunset.
How subjects answered
images/wan22/texture_attribute_binding/3.png
images/seeDream3/texture_attribute_binding/3.png
A circular clock with square numbers on the face, mounted on a triangular wall. The background is brightly lit.
How subjects answered
images/seeDream3/shape_attribute_binding/17.png
images/sd2.1/shape_attribute_binding/17.png
A banner at a sports event that says 'Go Team!' with fans cheering in the background.
How subjects answered
images/seeDream3/text-based/18.png
images/sd2.1/text-based/18.png
An oil painting depicting an event in ancient Rome. A long table shows clear social hierarchy. The painting should capture the subtle interplay of emotions, social status, and unspoken tensions typical of the era.
How subjects answered
images/seeDream3/social_relations/0.png
images/sd2.1/social_relations/0.png
Subjects
17 subjects, ranked by mean response (accuracy) across this benchmark's items.