Multilingual
CMMLU
CMMLU (Chinese MMLU) — per-(model, question) correctness across 67 subjects, from OpenCompass CompassAcademic predictions (answer letter extracted from each model output vs gold).
11,580items
22subjects
CC-BY-NC-SA-4.0license
knowledgedomain
reasoningdomain
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1deepseek-r10.9093
- 2qwen-max-2025-01-250.8754
- 3deepseek-chat-v30.8582
- 4qwen2.5-72b-instruct0.8551
- 5MiniMax-Text-010.8461
- 6internlm3-8b-instruct0.8426
- 7qwen2.5-32b-instruct0.8356
- 8deepseek-v2_5-12100.8102
- 9qwen2.5-14b-instruct0.8063
- 10deepseek-v2_50.791
- 11qwen2.5-7b-instruct0.7813
- 12internlm2_5-20b-chat0.7765
- 13rlhf-lmdeploy0.762
- 14internlm2_5-7b-chat0.7567
- 15sft-s2-lmdeploy0.7462
- 16llama-3_3-70b-instruct0.739
- 17gemma3_27b_it0.6979
- 18gpt-4o-mini-2024-07-180.6901
- 19gemma-2-27b-it0.6321
- 20gemma-2-9b-it0.5793
- 21llama-3_1-8b-instruct0.5491
- 22llama-3_2-3b-instruct0.4578