Knowledge
MMLU-Pro
MMLU-Pro — 12K-question challenge MMLU variant across 14 subjects; per-question binary correctness + leaderboard per-category aggregates.
13,542items
48subjects
MITlicense
knowledgedomain
reasoningdomain
textmodality
item-level responses released
Saturation status: No
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1gemini-3.1-pro0.8733
- 2arx_03140.8297
- 3iask_pro0.8119
- 4arx_30.7824
- 5claude-3-5-sonnet-202410220.7764
- 6claude-3.5-sonnet0.7612
- 7gpt-4o-2024-08-060.7468
- 8sonnet-3.50.7284
- 9gpt4o(2024-05-13)0.7207
- 10gemini-1.5-pro-0020.7025
- 11opus0.6846
- 12deepseek-chat-v2_50.6583
- 13gemini-1.5-flash-0020.6409
- 14DeepSeek-Coder-V20.6363
- 15gpt-4o-mini0.6309
- 16Meta-Llama-3_1-70B-Instruct0.6282
- 17claude-3-5-haiku-202410220.6212
- 18flash0.5913
- 19Meta-Llama-3-70B-Instruct0.5558
- 20sonnet0.5512
- 21deepseek0.5248
- 22Meta-Llama-3_1-70B0.5247
- 23Meta-Llama-3-70B0.5201
- 24jamba-1.5-large0.4946
- 25Qwen1.5-110B0.492
- 26Qwen1.5-72B-Chat0.4715
- 27Meta-Llama-3_1-8B-Instruct0.4419
- 28Yi-34B0.4208
- 29mathstral-7B0.42
- 30Mixtral-8x7B-Instruct-v0.10.4189
- 31Phi-3-mini-4k-instruct0.4148
- 32Meta-Llama-3-8B-Instruct0.4012
- 33Mixtral-8x7B-v0.10.3988
- 34c4ai-command-r-v010.3659
- 35Meta-Llama-3_1-8B0.3649
- 36Llama-2-70b-hf0.3629