General
CharXiv
CharXiv: realistic chart understanding for multimodal LLMs, built from real arXiv figures. 1,000 validation charts, each with four descriptive questions (19 fixed templates) and one free-form reasoning question. Models are scored per (model, question); reasoning is exact-match, descriptive is GPT-4o-graded against a rubric.
5,000items
37subjects
CC-BY-SA-4.0license
reasoningdomain
imagemodality
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1Claude-3-5-Sonnet0.7964
- 2GPT-4o0.7698
- 3GPT-4V-Turbo0.7136
- 4InternVL2-Pro0.6949
- 5InternVL2-76B0.6792
- 6GPT-4o-mini0.6676
- 7Gemini-1-5-Pro0.6624
- 8Claude-3-Sonnet0.654
- 9Claude-3-Opus0.6329
- 10Claude-3-Haiku0.5846
- 11InternVL2-26B0.566
- 12Ovis1.5-Gemma2-9B0.5582
- 13Phi-3-Vision0.547
- 14Ovis1.5-Llama3-8B0.5392
- 15Cambrian-34B0.5324
- 16InternVL-Chat-V1-50.5265
- 17MiniCPM-V2-50.5245
- 18Reka-Flash0.5051
- 19Reka-Core0.5028
- 20InternLM-XComposer2-4KHD0.4872
- 21Gemini-1.0-Pro-Vision0.4807
- 22Mini-Gemini-HD-Yi-34B0.4716
- 23LLaVA-1-6-Yi-34B0.4536
- 24DeepSeek-VL0.4008
- 25Mini-Gemini-HD-LLaMA3-8B0.3938
- 26Qwen-VL-Max0.3812
- 27IDEFICS-2-Chatty0.3682
- 28VILA-1-5-40B0.3574
- 29InternLM-XComposer20.3476
- 30MiniCPM-V20.324
- 31Reka-Edge0.3163
- 32LLaVA-1-6-Mistral-7B0.3116
- 33IDEFICS-20.2986
- 34SPHINX-V2-13B0.2744
- 35MoAI0.2647
- 36Qwen-VL-Plus0.2635