Knowledge

MMLU-Pro

MMLU-Pro poses multiple choice exam questions across fields such as law, physics and psychology, using a longer list of answer options and harder questions than the original MMLU.

13,542items

48subjects

MITlicense

knowledgedomain

reasoningdomain

textmodality

item-level responses released

Saturation status: No

Full data on Hugging Face Back to the gallery

Paper and credit

2024

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (opens in a new tab)

Original source

Authors

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni +13 moreShow all

01Yubo Wang
02Xueguang Ma
03Ge Zhang
04Yuansheng Ni
05Abhranil Chandra
06Shiguang Guo
07Weiming Ren
08Aaran Arulraj
09Xuan He
10Ziyan Jiang
11Tianle Li
12Max Ku
13Kai Wang
14Alex Zhuang
15Rongqi Fan
16Xiang Yue
17Wenhu Chen

Citation

BibTeX citationShow

BibTeX

1@misc{wang2024mmluprorobustchallengingmultitask,
2      title={MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark},
3      author={Yubo Wang and Xueguang Ma and Ge Zhang and Yuansheng Ni and Abhranil Chandra and Shiguang Guo and Weiming Ren and Aaran Arulraj and Xuan He and Ziyan Jiang and Tianle Li and Max Ku and Kai Wang and Alex Zhuang and Rongqi Fan and Xiang Yue and Wenhu Chen},
4      year={2024},
5      eprint={2406.01574},
6      archivePrefix={arXiv},
7      primaryClass={cs.CL},
8      url={https://arxiv.org/abs/2406.01574},
9}

Response matrix

leaderboard per-category aggregates

Rasch analysis p = σ(θ − z)

564,750 responses, 80/20 split over cells · 48 subjects · 13,542 items

AUC train

0.903

AUC test

0.889

Loading session strips…

Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect