Knowledge

MMLU

Published MMLU likelihood-based accuracy observations from the preserved 150-model Open LLM Leaderboard v1 panel.

14,015items

150subjects

MMLU questions and reference harness: MIT. Leaderboard observations: public original releases; no separate data license is declared in the captured dataset cards.license

knowledgedomain

reasoningdomain

textmodality

item-level responses released

Saturation status: No

Full data on Hugging Face Back to the gallery

Paper and credit

2021

Measuring Massive Multitask Language Understanding (opens in a new tab)

Original source

Authors

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou +3 moreShow all

01Dan Hendrycks
02Collin Burns
03Steven Basart
04Andy Zou
05Mantas Mazeika
06Dawn Song
07Jacob Steinhardt

Citation

BibTeX citationShow

BibTeX

1@inproceedings{hendrycks2021mmlu,
2  title={Measuring Massive Multitask Language Understanding},
3  author={Hendrycks, Dan and Burns, Collin and Basart, Steven and Zou, Andy and Mazeika, Mantas and Song, Dawn and Steinhardt, Jacob},
4  booktitle={ICLR},
5  year={2021},
6}

Response matrix

The 14,042 evaluated questions map to 13,937 distinct items (identical question texts share one item). Subject is recorded in test_condition. Model panel capped to 150.

Loading response matrix…

Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect