Stanford UniversityAI Measurement ScienceAIMS — AI Measurement Science

Measurement Data Bank

A curated data bank of AI evaluation results, standardized for measurement.

Dataset on Hugging Face

The data layer

measurement-db standardizes evaluation results from 201 public benchmarks into (model × item) response matrices, the raw observations that item response theory and other psychometric models are estimated from. It is the data layer beneath torch_measure.

Maintained by Nhi Truong, Sang Truong, and Sanmi Koyejo

01

201

Benchmarks curated

02

2.1M

Unique evaluation items

03

64.1M

Model–item responses

01What it is

Coverage spans reasoning, coding, agentic, multimodal, safety, and human-preference benchmarks. Every benchmark ships a reproducible build script, and each response carries its model, item, benchmark, and test condition.

Browse the dataset

02Coverage

What the bank measures, and how deeply: the frontier models it tracks, and the domains its benchmarks cover.

Ranked by items asked

Frontier Model Coverage

A curated set of frontier models ranked by the number of distinct benchmark items they have been evaluated on.

#ModelsItems asked
  1. 1Microsoft

    Microsoft

    Phi-4

    45,415
  2. 2Meta

    Meta

    Llama 4 Maverick

    37,096
  3. 3Salesforce

    Salesforce

    xGen-MM

    7,541
  4. 4Tencent

    Tencent

    Hunyuan Standard Vision

    7,433
  5. 5Cohere

    Cohere

    Command A

    7,000
  6. 6Google

    Google

    Gemini 3.1 Pro

    3,422
  7. 7IBM

    IBM

    Granite 4.0 Micro

    2,944
  8. 8OpenAI

    OpenAI

    GPT-5.5

    2,276
  9. 9Mistral

    Mistral

    Large 3

    2,070
  10. 10DeepSeek

    DeepSeek

    V4 Pro

    1,800
  11. 11Anthropic

    Anthropic

    Claude Opus 4.8

    902
  12. 12Alibaba

    Alibaba

    Qwen3.7 Max

    41
  13. 13Amazon

    Amazon

    Nova 2 Lite

    40
  14. 14M

    Moonshot

    Kimi K2.7

    40
  15. 15xAI

    xAI

    Grok 4.3

    40

Ranked by items covered

Domain Coverage Assessment

Benchmarks that span multiple domains count toward each, so the totals overlap rather than partition the bank.

#DomainItems
  1. 1

    Reasoning

    69 benchmarks

    1,176,697
  2. 2

    Knowledge

    38 benchmarks

    759,312
  3. 3

    Safety

    44 benchmarks

    444,135
  4. 4

    General

    10 benchmarks

    398,000
  5. 5

    Mathematics

    24 benchmarks

    296,542
  6. 6

    NLP Tasks

    9 benchmarks

    286,828
  7. 7

    ML Engineering

    5 benchmarks

    271,701
  8. 8

    Science

    18 benchmarks

    265,723
  9. 9

    Preference

    10 benchmarks

    205,834
  10. 10

    Multilingual

    10 benchmarks

    136,069
  11. 11

    Medicine

    31 benchmarks

    130,464
  12. 12

    Law

    2 benchmarks

    86,374
  13. 13

    Software Engineering

    26 benchmarks

    63,560
  14. 14

    Agents & Tool Use

    24 benchmarks

    36,280
  15. 15

    Cultural

    3 benchmarks

    24,864

Average score per domain, every model

Model × Domain Scores

Every model’s average score across each benchmark domain. A hatched cell means the model was never measured in that domain.

Model
  1. 72
    79
    72
    79
    81
    81
    68
    79
    48
    62
    65
    80
  2. 77
    73
    77
    76
    77
    77
    77
    77
    77
  3. 77
    73
    77
    77
    77
    77
    77
    77
    77
  4. 78
    94
    71
    97
    65
    82
    64
    81
  5. 76
    73
    76
    76
    76
    76
    76
    76
    76
  6. 82
    74
    68
    75
    75
    75
    63
    75
    63
    62
    73
  7. 76
    71
    76
    74
    76
    77
    76
    76
    76
  8. 76
    71
    74
    75
    75
    76
    74
    75
    75
  9. 70
    72
    76
    76
    76
    70
    76
    76
    76
  10. 77
    63
    75
    75
    75
    77
    75
    75
    75
  11. 81
    79
    81
    81
    81
    81
    81
    81
    25
    25
    81
  12. 61
    64
    77
    80
    83
    83
    66
    80
    60
    78
    22
    40
    51
    79
  13. 75
    65
    73
    73
    73
    75
    73
    73
    73
  14. 73
    69
    73
    73
    73
    73
    73
    73
    73
  15. 69
    67
    74
    74
    74
    69
    74
    74
    74
  16. 53
    63
    75
    67
    79
    79
    71
    62
    74
    71
    54
    37
    58
    73
  17. 88
    85
    77
    83
    77
    83
    49
  18. 76
    59
    73
    73
    73
    76
    73
    70
    73
  19. 43
    66
    67
    77
    78
    78
    68
    77
    76
    81
    17
    77
  20. 71
    76
    76
    70
    83
    70
    51
    56
    45
    97
  21. 87
    67
    61
    72
    71
    72
    71
    69
    50
    70
    68
    50
    23
    67
    62
  22. 41
    69
    79
    68
    82
    82
    73
    69
    82
    77
    68
    0
    40
    74
  23. 86
    61
    70
    78
    78
    79
    63
    79
    92
    49
    13
    1
    79
    52
    70
    58
  24. 84
    68
    78
    80
    83
    83
    67
    82
    54
    6
    0
    82
    56
    77
  25. 78
    79
    75
    80
    80
    80
    50
    66
    39
    36
    78
  26. 75
    58
    69
    68
    69
    75
    69
    75
    69
  27. 28
    57
    71
    73
    74
    74
    57
    73
    74
    68
    63
    72
  28. 69
    68
    69
    69
    69
    69
    69
    69
    69
  29. 85
    90
    74
    81
    74
    72
    42
  30. 67
    69
    73
    73
    73
    69
    73
    73
    40
    38
    63
    70
  31. 90
    88
    76
    79
    79
    79
    59
    62
    60
    12
    12
    39
    97
  32. 54
    72
    82
    85
    86
    86
    72
    83
    63
    35
    40
    30
    45
    80
    19
  33. 67
    58
    70
    70
    70
    67
    70
    68
    70
  34. 96
    94
    96
    39
  35. 67
    80
    73
    71
    85
    70
    52
    60
    31
  36. 29
    57
    69
    71
    71
    71
    59
    70
    66
    57
    63
    64
  37. 75
    57
    67
    67
    67
    75
    67
    75
    21
    67
  38. 76
    65
    70
    72
    70
    65
    72
    66
    52
    32
  39. 66
    63
    65
    65
    65
    66
    65
    65
    65
  40. 88
    91
    88
    54
  41. 66
    61
    65
    65
    65
    67
    65
    65
    65
  42. 29
    57
    55
    59
    69
    69
    67
    60
    69
    67
    69
    59
    70
  43. 78
    93
    69
    97
    35
  44. 53
    57
    72
    78
    78
    78
    54
    73
    81
    20
    52
    19
    81
  45. 66
    57
    65
    65
    65
    66
    65
    66
    65
  46. 70
    66
    72
    73
    75
    75
    59
    73
    55
    0
    71
    32
    73
  47. 41
    60
    52
    69
    69
    69
    60
    69
    63
    57
    61
    69
  48. 87
    89
    87
    50
  49. 65
    54
    66
    66
    66
    64
    66
    63
    66
  50. 88
    61
    71
    97
    48
  51. 58
    63
    66
    66
    66
    58
    66
    66
    66
  52. 62
    65
    64
    62
    64
    64
    64
    64
    64
  53. 24
    53
    66
    69
    69
    69
    53
    69
    69
    64
    63
    67
  54. 66
    59
    64
    64
    64
    66
    64
    64
    64
  55. 61
    65
    64
    61
    64
    64
    64
    64
    64
  56. 48
    66
    59
    71
    75
    75
    73
    66
    75
    71
    21
    25
    40
    75
  57. 89
    91
    89
    38
  58. 79
    83
    79
    63
    63
    47
  59. 88
    91
    88
    39
  60. 85
    91
    85
    45
  61. 88
    92
    88
    38
  62. 88
    79
    88
    48
  63. 98
    98
  64. 88
    90
    88
    37
  65. 98
    98
  66. 91
    90
    91
    29
  67. 89
    90
    89
    31
  68. 96
    96
  69. 96
    96
  70. 81
    81
    81
  71. 63
    57
    63
    63
    63
    63
    63
    63
    56
    63
  72. 64
    56
    63
    63
    63
    64
    63
    63
    63
  73. 86
    82
    74
  74. 92
    96
  75. 89
    87
    89
    30
  76. 86
    89
    86
    31
  77. 81
    85
    81
    46
  78. 93
    93
  79. 81
    86
    81
    43
  80. 93
    93
  81. 84
    85
    84
    39
  82. 85
    88
    85
    34
  83. 93
    93
  84. 87
    89
    87
    29
  85. 79
    73
    71
    72
    66
    46
    47
    49
  86. 87
    87
    87
    29
  87. 87
    88
    87
    28
  88. 8
    69
    75
    69
    75
    75
    74
    69
    42
  89. 81
    87
    81
    40
  90. 92
    92
  91. 92
    92
  92. 91
    91
  93. 84
    72
    84
    48
  94. 86
    86
    86
    30
  95. 88
    88
    44
    68
  96. 87
    87
    87
    27
  97. 79
    81
    79
    47
  98. 78
    78
    78
  99. 78
    78
    77
  100. 86
    87
    86
    28
  101. 90
    90
  102. 90
    90
  103. 90
    90
  104. 80
    77
    76
  105. 90
    90
  106. 57
    60
    66
    62
    66
    58
    66
    51
    66
  107. 90
    90
  108. 85
    89
    85
    26
  109. 86
    84
    86
    28
  110. 89
    71
    77
    76
    24
  111. 89
    86
    89
    34
    38
  112. 69
    86
    67
    74
    64
    29
  113. 84
    88
    84
    26
  114. 80
    88
    80
    34
  115. 98
    77
  116. 84
    88
    84
    26
  117. 88
    88
  118. 85
    90
    85
    21
  119. 85
    87
    85
    24
  120. 77
    80
    77
    46
  121. 81
    85
    81
    33
  122. 87
    87
  123. 61
    61
    61
    61
    61
    61
    61
    61
    61
  124. 83
    86
    83
    26
  125. 84
    87
    84
    24
  126. 78
    74
    73
  127. 62
    51
    62
    62
    62
    62
    62
    60
    62
  128. 83
    86
    83
    27
  129. 86
    86
  130. 83
    83
    83
    28
  131. 96
    75
  132. 86
    86
  133. 83
    72
    70
  134. 83
    85
    83
    26
  135. 85
    85
  136. 82
    86
    82
    27
  137. 85
    85
  138. 82
    87
    82
    25
  139. 61
    50
    62
    61
    62
    62
    62
    60
    62
  140. 82
    86
    82
    26
  141. 82
    86
    82
    24
  142. 85
    85
  143. 84
    84
  144. 88
    82
    88
    38
    31
  145. 79
    85
    79
    32
  146. 86
    91
    86
    11
  147. 81
    87
    81
    24
  148. 82
    84
    82
    26
  149. 74
    74
    73
  150. 81
    69
    81
    44
  151. 83
    83
    83
    25
  152. 74
    68
    55
    64
    64
    64
    66
    64
    64
    0
    51
    63
  153. 81
    86
    81
    22
  154. 73
    73
    73
  155. 35
    62
    58
    72
    72
    72
    59
    72
    51
    68
    59
    8
    48
    68
  156. 82
    82
  157. 82
    82
  158. 78
    83
    78
    31
  159. 75
    81
    75
    38
  160. 81
    81
  161. 81
    81
  162. 81
    81
  163. 71
    71
    66
    69
    66
    31
  164. 82
    86
    82
    18
  165. 79
    88
    79
    22
  166. 71
    71
    71
  167. 78
    85
    78
    26
  168. 77
    74
    77
    44
    45
    53
    56
  169. 79
    68
    77
    77
    77
    79
    77
    77
    0
    0
    0
    77
  170. 80
    80
  171. 71
    71
    71
  172. 81
    83
    81
    20
  173. 85
    83
    85
    33
    33
  174. 79
    85
    79
    23
  175. 80
    80
  176. 61
    52
    59
    58
    59
    61
    59
    63
    59
  177. 84
    70
    84
    25
  178. 61
    47
    61
    61
    61
    61
    61
    59
    61
  179. 79
    79
  180. 75
    82
    75
    32
  181. 42
    68
    75
    64
    75
    75
    59
    35
    62
    41
    56
    43
    42
    57
  182. 79
    83
    79
    21
  183. 78
    78
  184. 68
    76
    65
  185. 78
    82
    78
    24
  186. 78
    78
  187. 74
    82
    74
    32
  188. 73
    82
    73
    32
  189. 69
    69
    69
  190. 55
    62
    70
    65
    76
    78
    73
    55
    72
    64
    53
    30
    27
    38
    69
    11
  191. 74
    81
    74
    31
  192. 83
    70
    83
    26
  193. 54
    100
    54
    54
  194. 75
    78
    75
    33
  195. 74
    83
    74
    29
  196. 77
    83
    77
    23
  197. 77
    77
  198. 51
    93
    63
  199. 77
    77
  200. 72
    82
    72
    34
  201. 100
  202. 100
  203. 100
  204. 100
  205. 100
  206. 100
  207. 100
  208. 100
  209. 100
  210. 100
  211. 100
  212. 100
  213. 80
    69
    80
    30
  214. 76
    76
  215. 100
    53
  216. 74
    78
    74
    33
  217. 76
    76
  218. 67
    71
    67
  219. 98
  220. 76
    76
  221. 39
    61
    52
    61
    68
    67
    68
  222. 68
    68
    68
  223. 76
    78
    76
    26
  224. 75
    75
  225. 76
    83
    76
    21
  226. 76
    85
    76
    19
  227. 77
    82
    77
    21
  228. 77
    78
    77
    24
  229. 99
    51
  230. 75
    75
  231. 92
    96
    15
  232. 89
    90
    89
    14
    27
  233. 75
    75
  234. 75
    75
  235. 75
    75
  236. 67
    67
    67
  237. 83
    78
    89
    39
    59
    36
    30
  238. 74
    74
  239. 74
    74
  240. 75
    82
    75
    21
  241. 78
    77
    78
    21
  242. 58
    58
    58
    58
    58
    58
    58
    58
    58
  243. 25
    72
    74
    72
    72
    74
    76
    66
    50
    44
    39
    14
  244. 73
    73
  245. 73
    73
  246. 73
    73
  247. 51
    56
    60
    60
    60
    51
    60
    60
    60
  248. 66
    66
    66
  249. 73
    73
  250. 78
    68
    78
    27
  251. 75
    80
    75
    22
  252. 73
    73
  253. 85
    88
    85
    3
    44
  254. 91
  255. 74
    83
    74
    20
  256. 91
  257. 69
    77
    69
    35
  258. 59
    42
    59
    59
    59
    59
    59
    58
    59
  259. 69
    78
    69
    33
  260. 65
    65
    65
  261. 74
    66
    74
    35
  262. 72
    72
  263. 90
  264. 78
    83
    78
    11
  265. 79
    79
    38
  266. 71
    71
  267. 71
    71
  268. 78
    60
    78
    32
  269. 89
    98
    28
    34
  270. 75
    80
    75
    19
  271. 76
    85
    76
    12
  272. 64
    64
    67
  273. 70
    76
    70
    33
  274. 88
  275. 52
    91
    52
  276. 75
    80
    75
    17
  277. 70
    70
  278. 26
    48
    54
    65
    65
    65
    48
    65
    65
    52
    65
  279. 70
    70
  280. 70
    70
  281. 87
  282. 49
    30
    69
    77
    75
  283. 60
    46
    56
    59
    59
    63
    59
    55
    48
    59
  284. 70
    70
  285. 70
    70
  286. 71
    81
    71
    23
  287. 86
  288. 86
  289. 56
    62
    57
    56
    57
    56
    57
    55
    57
  290. 70
    70
  291. 86
  292. 70
    70
  293. 83
    85
    83
    24
    24
  294. 69
    69
  295. 85
  296. 69
    69
  297. 69
    69
  298. 64
    64
    64
  299. 69
    82
    69
    23
  300. 85
    55
    51
  301. 84
  302. 74
    80
    69
    21
  303. 69
    69
  304. 68
    68
  305. 84
  306. 49
    53
    87
    55
  307. 86
    89
    86
    9
    26
  308. 83
  309. 53
    55
    72
    72
    72
    25
  310. 71
    76
    71
    24
  311. 72
    72
    72
    71
    31
    31
  312. 82
    79
    82
    26
    27
  313. 83
  314. 99
    73
    35
    35
  315. 68
    68
  316. 86
    63
    37
    37
    86
    39
  317. 82
  318. 81
    62
    81
    17
  319. 60
    65
    60
    60
    60
    60
    60
    58
    16
    60
  320. 82
  321. 63
    79
    63
    36
  322. 73
    82
    73
    13
  323. 78
    57
  324. 58
    39
    59
    59
    59
    58
    59
    58
    59
  325. 67
    67
  326. 85
    53
    49
  327. 80
  328. 78
    67
    78
    16
  329. 67
    67
  330. 69
    79
    69
    23
  331. 63
    74
    66
    36
  332. 66
    66
  333. 79
  334. 71
    76
    71
    21
  335. 66
    66
  336. 47
    60
    50
    36
    91
    64
    51
  337. 79
  338. 73
    78
    73
    14
  339. 80
    52
  340. 79
  341. 78
  342. 74
    77
    74
    14
  343. 78
  344. 66
    78
    66
    26
  345. 72
    84
    72
    10
  346. 78
  347. 88
    43
  348. 72
    77
    72
    15
  349. 71
    75
    71
    19
  350. 65
    65
  351. 47
    55
    92
    43
  352. 70
    76
    70
    20
  353. 63
    76
    63
    33
  354. 64
    75
    64
    32
  355. 75
  356. 75
  357. 60
    60
    60
  358. 63
    63
  359. 70
    80
    70
    13
  360. 64
    76
    64
    29
  361. 63
    63
  362. 73
  363. 73
  364. 60
    60
    60
  365. 60
    60
    60
  366. 66
    62
    68
    70
    71
    71
    63
    70
    59
    0
    70
    21
    69
    4
  367. 63
    96
    63
    63
    40
    14
  368. 59
    28
    59
    59
    59
    59
    59
    57
    59
  369. 63
    63
  370. 63
    63
  371. 72
  372. 69
    78
    69
    16
  373. 72
  374. 79
    62
    72
    19
  375. 65
    81
    65
    20
  376. 71
  377. 70
    56
    48
    53
    53
    53
    55
    53
    53
    52
  378. 76
    64
    76
    16
  379. 71
  380. 62
    62
  381. 63
    61
  382. 62
    62
  383. 71
  384. 67
    80
    67
    14
  385. 70
  386. 70
  387. 72
    70
    72
    15
  388. 69
  389. 69
  390. 65
    41
    59
    59
    59
    64
    59
    55
    24
    53
    61
  391. 69
  392. 70
    75
    70
    13
  393. 69
  394. 63
    58
    55
  395. 61
    61
  396. 69
  397. 68
  398. 93
    59
    66
    10
  399. 68
  400. 68
  401. 67
    54
    53
  402. 68
  403. 68
  404. 68
  405. 68
  406. 58
    58
    58
  407. 67
  408. 65
    80
    65
    16
  409. 56
    91
    58
    19
    56
  410. 67
  411. 84
    49
    69
    11
    67
  412. 66
  413. 60
    62
    68
    66
    68
    61
    68
    50
    4
    19
    68
  414. 59
    59
  415. 66
  416. 66
  417. 60
    46
    60
    59
  418. 53
    54
    53
    53
    53
    53
    53
    54
    66
    53
  419. 59
    59
  420. 64
  421. 68
    74
    91
    78
    44
    61
    42
    38
    42
    38
    20
  422. 64
  423. 64
  424. 59
    59
  425. 64
  426. 64
  427. 64
  428. 58
    58
  429. 64
  430. 63
  431. 63
  432. 63
  433. 58
    58
  434. 63
  435. 63
  436. 65
    76
    65
    15
  437. 69
    77
    69
    7
  438. 69
    54
    69
    30
  439. 66
    75
    66
    13
  440. 62
  441. 62
  442. 62
  443. 61
  444. 54
    57
    54
    54
    54
    54
    54
    52
    54
  445. 61
  446. 61
  447. 77
    89
    77
    22
    22
    39
  448. 61
  449. 61
  450. 61
  451. 60
  452. 60
  453. 60
  454. 98
    71
    25
    25
  455. 64
    54
    48
  456. 60
  457. 65
    54
    65
    35
  458. 59
  459. 66
    77
    66
    10
  460. 59
  461. 59
  462. 56
    56
  463. 59
  464. 59
  465. 59
  466. 59
  467. 55
    55
    55
  468. 59
  469. 46
    67
    52
  470. 59
  471. 58
  472. 58
  473. 55
    45
    55
    55
    55
    55
    55
    54
    55
  474. 58
  475. 65
    79
    65
    7
  476. 58
  477. 57
  478. 57
  479. 57
  480. 57
  481. 46
    45
    71
    46
    71
    44
  482. 57
  483. 57
  484. 55
    55
  485. 57
  486. 57
  487. 35
    64
    37
    69
    65
  488. 57
  489. 57
  490. 56
  491. 55
    55
  492. 70
    58
    35
  493. 56
  494. 62
    75
    62
    17
  495. 56
  496. 56
  497. 56
  498. 56
    76
    56
    27
  499. 56
  500. 46
    58
    58
  501. 64
    79
    64
    8
  502. 56
  503. 55
  504. 55
  505. 55
  506. 55
  507. 54
    54
    54
  508. 55
  509. 55
    42
    55
    55
    55
    55
    55
    53
    55
  510. 58
    69
    58
    29
  511. 49
    59
  512. 53
    53
    53
  513. 54
    54
  514. 53
    53
    53
  515. 54
  516. 54
  517. 53
    55
    80
    54
    63
    70
    36
    55
    24
    41
  518. 54
  519. 53
  520. 53
    53
  521. 53
  522. 64
    78
    64
    8
  523. 45
    57
    57
  524. 22
    43
    28
    95
    56
    77
    50
  525. 53
    53
  526. 53
    53
  527. 53
    58
    53
    53
    53
    53
    53
    51
    53
  528. 53
  529. 53
  530. 52
  531. 52
  532. 72
    49
    72
    18
  533. 52
  534. 68
    53
    68
    22
  535. 52
    52
  536. 52
  537. 52
  538. 52
  539. 52
  540. 52
  541. 22
    48
    49
    50
    58
    62
    61
    50
    62
    63
    45
    55
    62
  542. 52
    52
  543. 56
    28
    56
    56
    56
    56
    56
    55
    56
  544. 52
    52
  545. 59
    72
    59
    20
  546. 51
  547. 55
    47
    53
    53
    53
    55
    53
    53
    53
  548. 52
    52
  549. 61
    76
    61
    11
  550. 51
  551. 52
    52
  552. 50
  553. 50
  554. 50
  555. 50
  556. 50
  557. 50
  558. 50
  559. 60
    76
    60
    12
  560. 60
    74
    60
    15
  561. 76
    76
    5
  562. 49
  563. 49
  564. 49
  565. 63
    39
  566. 48
  567. 68
    48
    38
  568. 48
  569. 48
  570. 48
  571. 51
    51
  572. 48
  573. 48
  574. 48
  575. 78
    23
  576. 47
  577. 47
  578. 47
  579. 47
  580. 47
  581. 46
    60
    41
    60
  582. 50
    50
  583. 50
    50
  584. 47
  585. 46
  586. 46
  587. 49
    75
    49
    31
  588. 49
    49
  589. 51
    51
    51
  590. 60
    76
    60
    9
  591. 55
    33
    55
    55
    55
    55
    55
    53
    55
  592. 61
    77
    61
    5
  593. 45
  594. 44
  595. 44
  596. 59
    75
    59
    11
  597. 50
    50
    50
  598. 43
  599. 43
  600. 43
  601. 81
    58
    10
  602. 90
    57
    65
    79
    11
    7
  603. 58
    74
    58
    12
  604. 35
    49
    49
    69
  605. 47
    48
  606. 58
    76
    58
    10
  607. 47
    47
  608. 37
    58
  609. 41
  610. 49
    49
    49
  611. 57
    77
    57
    9
  612. 59
    65
    59
    16
  613. 40
  614. 57
    76
    57
    10
  615. 47
    47
  616. 47
    47
  617. 57
    23
    51
    68
  618. 46
    46
  619. 39
  620. 83
    69
    83
    15
    38
    18
  621. 46
    46
  622. 39
  623. 39
  624. 39
  625. 46
    46
  626. 39
  627. 52
    47
    52
    52
    52
    52
    52
    51
    52
  628. 58
    75
    58
    7
  629. 62
    68
    50
    62
    56
    56
    69
    63
    70
    25
    29
    13
    54
  630. 38
  631. 46
    46
  632. 46
    46
  633. 38
  634. 38
  635. 78
    79
    78
    0
    15
  636. 38
  637. 37
  638. 55
    62
    55
    24
  639. 57
    67
    57
    15
  640. 37
  641. 31
    18
    97
    46
    53
    58
  642. 45
    45
  643. 57
    75
    57
    8
  644. 58
    17
    68
  645. 36
  646. 45
    45
  647. 45
    45
  648. 36
  649. 36
  650. 52
    45
    52
    52
    52
    52
    52
    51
    52
  651. 51
    74
    51
    19
  652. 45
    45
  653. 59
    66
    59
    11
  654. 39
    58
    57
    57
    57
    39
    57
    39
    57
  655. 36
  656. 35
  657. 35
  658. 44
    44
  659. 35
  660. 57
    69
    57
    10
  661. 34
  662. 67
    76
    67
    18
    18
  663. 33
  664. 33
  665. 33
  666. 49
    40
    51
  667. 33
  668. 38
    37
    99
    38
    33
  669. 32
  670. 43
    43
  671. 43
    43
  672. 42
    42
  673. 54
    69
    54
    15
  674. 74
    59
    5
  675. 31
  676. 42
    42
  677. 56
    73
    56
    6
  678. 31
  679. 31
  680. 42
    42
  681. 30
  682. 41
    41
  683. 29
  684. 48
    49
    49
    49
    48
  685. 29
  686. 49
    49
    48
    47
    49
  687. 29
  688. 28
  689. 41
    41
  690. 41
    41
  691. 51
    43
    40
  692. 28
  693. 45
    45
    45
  694. 53
    68
    53
    14
  695. 27
  696. 40
    40
  697. 58
    39
    36
  698. 51
    45
    51
    51
    51
    51
    51
    50
    51
  699. 27
  700. 26
  701. 26
  702. 26
  703. 43
    57
    59
    26
  704. 50
    43
    38
  705. 46
    46
    50
    51
    46
  706. 25
  707. 25
  708. 25
  709. 24
  710. 38
    38
  711. 38
    38
  712. 49
    44
    37
  713. 56
    20
  714. 23
  715. 50
    54
    50
    50
    50
    50
    50
    48
    50
  716. 23
  717. 23
  718. 38
    38
  719. 2
    68
    68
    68
    74
    47
    15
  720. 18
    18
    93
  721. 22
  722. 43
    43
    43
  723. 37
    37
  724. 45
    45
    37
  725. 42
    42
    42
  726. 20
  727. 49
    49
    49
    49
    49
    49
    49
    49
    49
  728. 49
    54
    49
    49
    49
    49
    49
    48
    49
  729. 73
    78
    73
    1
    7
  730. 45
    48
    42
    48
    48
    48
    49
    48
    56
    70
    48
  731. 20
  732. 53
    71
    51
    21
    81
    8
  733. 19
  734. 53
    48
    53
    24
  735. 52
    30
    52
    52
    52
    52
    52
    50
    52
  736. 19
  737. 19
  738. 18
  739. 61
    76
    61
    21
    11
  740. 43
    46
    35
  741. 39
    41
    42
  742. 44
    44
    35
  743. 49
    71
    49
    6
  744. 44
    43
    34
  745. 15
  746. 39
    48
    35
  747. 34
    34
  748. 46
    53
    46
    28
  749. 80
    98
    25
    12
    12
  750. 14
  751. 14
  752. 51
    34
    51
    51
    51
    51
    51
    49
    51
  753. 75
    42
    28
    28
  754. 12
  755. 39
    39
    39
  756. 32
    44
    60
    59
    60
    32
    60
    30
    60
  757. 11
  758. 51
    28
    51
    51
    51
    51
    51
    50
    51
  759. 31
    31
    83
    25
  760. 23
    77
    23
    77
    23
  761. 10
  762. 10
  763. 10
  764. 39
    39
    39
  765. 31
    31
  766. 9
  767. 9
  768. 31
    31
  769. 41
    62
    41
    25
  770. 11
    91
    57
    63
    63
    56
    63
    63
    10
    10
  771. 9
  772. 37
    45
    32
  773. 8
  774. 8
  775. 48
    51
    48
    48
    48
    48
    48
    47
    48
  776. 53
    42
    53
    55
    55
    52
    55
    52
    49
    19
    55
  777. 8
  778. 7
  779. 42
    66
    42
    15
  780. 46
    59
    46
    14
  781. 49
    96
    49
    55
    23
    1
  782. 29
    29
  783. 28
    43
    38
    95
    52
    15
  784. 5
  785. 5
  786. 5
  787. 37
    41
    33
  788. 4
  789. 38
    64
    38
    24
  790. 41
    41
    41
    41
  791. 42
    60
    42
    20
  792. 4
  793. 34
    42
    34
  794. 17
    17
    100
    17
    66
  795. 64
    43
    43
    13
  796. 36
    43
    30
  797. 3
  798. 48
    48
    48
    48
    48
    48
    48
    48
    48
  799. 56
    70
    56
    21
    11
  800. 47
    58
    47
    9
  801. 27
    27
  802. 47
    48
    47
    47
    47
    47
    47
    47
    47
  803. 68
    20
    20
  804. 47
    47
    47
    47
    47
    47
    47
    47
    47
  805. 36
    36
    36
  806. 38
    41
    28
  807. 27
    27
  808. 29
    29
    83
    26
    44
    54
  809. 0
  810. 35
    35
    35
  811. 0
  812. 46
    42
    19
  813. 47
    46
    47
    47
    47
    47
    47
    51
    47
  814. 38
    40
    28
  815. 41
    57
    41
    19
  816. 22
    45
    50
    55
    54
    54
    43
    55
    58
    50
    46
    51
  817. 42
    50
    53
    58
    58
    42
    58
    37
    26
    55
  818. 34
    42
    29
  819. 35
    40
    29
  820. 35
    41
    29
  821. 26
    71
    26
    62
    26
  822. 27
    42
    35
  823. 34
    40
    29
  824. 36
    32
    36
  825. 34
    34
    34
  826. 46
    57
    46
    46
    46
    46
    46
    44
    46
  827. 34
    40
    28
  828. 35
    57
    47
    57
    56
    10
  829. 33
    38
    30
  830. 33
    39
    29
  831. 33
    33
    33
  832. 35
    30
    35
  833. 23
    23
  834. 34
    37
    28
  835. 34
    37
    27
  836. 32
    38
    28
  837. 32
    36
    29
  838. 32
    11
  839. 48
    45
    48
    48
    48
    48
    48
    32
    48
  840. 32
    37
    27
  841. 75
    11
    11
  842. 46
    46
    46
    46
    46
    46
    46
    46
    46
  843. 32
    37
    27
  844. 28
    37
    46
    37
  845. 37
    37
    37
    37
  846. 46
    46
    46
    46
    46
    46
    46
    46
    46
  847. 30
    32
    33
  848. 31
    31
    31
  849. 46
    45
    46
    46
    46
    46
    46
    45
    46
  850. 20
    20
  851. 33
    56
    33
    24
  852. 44
    57
    44
    44
    44
    44
    44
    44
    44
  853. 31
    31
    31
  854. 31
    35
    27
  855. 34
    24
    40
    48
  856. 31
    31
    31
  857. 45
    45
    45
    45
    45
    45
    45
    44
    45
  858. 57
    68
    53
    59
    59
    56
    59
    56
    1
    33
    7
    59
  859. 21
    53
    53
    55
    56
    56
    54
    36
    72
    43
    11
    63
    48
    55
    49
  860. 32
    3
  861. 16
    19
  862. 39
    49
    39
    13
  863. 30
    31
    25
  864. 29
    30
    26
  865. 28
    28
    28
  866. 29
    30
    25
  867. 30
    29
    24
  868. 28
    30
    25
  869. 29
    27
    26
  870. 29
    29
    24
  871. 14
    14
  872. 46
    37
    46
    46
    46
    46
    46
    44
    46
  873. 29
    28
    24
  874. 14
    14
  875. 17
    10
  876. 30
    50
    30
    23
  877. 27
    27
    25
  878. 27
    27
    23
  879. 25
    27
    26
  880. 25
    26
    26
  881. 19
    19
    19
    74
  882. 24
    27
    25
  883. 26
    25
    25
  884. 26
    26
    24
  885. 26
    26
    24
  886. 38
    62
    38
    21
    22
  887. 25
    26
    25
  888. 11
    11
  889. 25
    24
    26
  890. 25
    26
    25
  891. 23
    26
    26
  892. 25
    25
    24
  893. 25
    25
    24
  894. 45
    43
    45
    45
    45
    45
    45
    37
    45
  895. 25
    24
    25
  896. 13
    13
    48
  897. 25
    24
    24
  898. 27
    22
    23
  899. 24
    24
    24
  900. 53
    53
    53
    10
    10
  901. 23
    25
    24
  902. 25
    24
    24
  903. 26
    24
    23
  904. 26
    22
    24
  905. 25
    23
    24
  906. 24
    24
    24
  907. 24
    24
    24
  908. 24
    23
    24
  909. 24
    24
    23
  910. 25
    23
    23
  911. 25
    23
    23
  912. 25
    23
    22
  913. 27
    49
    27
    19
  914. 24
    23
    23
  915. 8
    8
  916. 25
    22
    22
  917. 23
    22
    24
  918. 18
    52
    49
    50
    53
    52
    51
    52
    59
    34
    31
    45
    53
  919. 22
    22
    22
  920. 55
    2
    73
    2
    67
    73
    79
    27
    53
    10
  921. 72
    24
    75
    56
    53
    65
    64
    49
    17
    47
    23
    40
    51
    23
    46
  922. 42
    46
    43
    43
    43
    42
    43
    41
    43
  923. 21
    21
    21
  924. 30
    31
    38
    38
    31
  925. 46
    52
    47
    47
    47
    46
    47
    46
    7
    47
  926. 13
    33
    13
  927. 19
    19
    19
  928. 1
    1
  929. 1
    1
  930. 1
    1
  931. 30
    30
    35
    37
    30
  932. 18
    18
    18
  933. 26
    30
    26
    22
  934. 31
    31
    31
    11
  935. 17
    17
    17
  936. 22
    8
    22
  937. 24
    28
    28
    36
    57
    36
  938. 36
    28
    28
    28
    36
  939. 36
    36
    36
    36
    36
    36
    36
    80
    36
  940. 48
    55
    46
    50
    50
    48
    50
    48
    27
    2
    48
  941. 39
    43
    41
    41
    41
    40
    41
    38
    41
  942. 41
    40
    41
    41
    41
    41
    41
    40
    41
  943. 40
    40
    40
    40
    40
    40
    40
    40
    40
  944. 27
    26
    27
    11
  945. 12
    12
    12
  946. 26
    26
    30
    28
    30
  947. 34
    13
    68
    15
    68
    31
    16
  948. 19
    19
    19
    31
  949. 29
    92
    29
    22
    19
    0
  950. 36
    52
    37
    37
    37
    36
    37
    36
    37
  951. 19
    23
    23
    32
    57
    32
  952. 23
    23
    30
    26
    30
  953. 38
    41
    38
    38
    38
    38
    38
    37
    38
  954. 6
    6
    6
  955. 39
    29
    38
    38
    38
    39
    38
    38
    38
  956. 17
    17
    17
    17
  957. 37
    37
    37
    37
    37
    37
    37
    37
    37
  958. 35
    52
    35
    35
    35
    35
    35
    34
    35
  959. 36
    39
    36
    36
    36
    36
    36
    35
    36
  960. 5
    38
    55
    55
    55
    54
    55
    53
    2
    5
  961. 33
    64
    33
    33
    33
    33
    33
    32
    33
  962. 10
    10
    10
    28
  963. 35
    35
    35
    35
    35
    35
    35
    35
    35
  964. 35
    35
    35
    35
    35
    35
    35
    35
    35
  965. 46
    0
    0
    0
  966. 4
    18
    4
    4
    60
  967. 5
    19
    5
    5
  968. 31
    37
    32
    32
    32
    31
    32
    30
    32
  969. 31
    32
    31
    31
    31
    31
    31
    31
    31
  970. 33
    33
    30
    33
    30
    30
    30
    30
    30
  971. 11
    14
    14
    22
    34
    22
  972. 28
    49
    28
    28
    28
    28
    28
    27
    28
  973. 29
    40
    29
    29
    29
    28
    29
    28
    29
  974. 1
    25
    24
    1
    1
  975. 2
    23
    22
    2
    2
  976. 0
    25
    25
    0
    0
  977. 1
    24
    24
    1
    1
  978. 0
    25
    25
    0
    0
  979. 29
    29
    29
    29
    29
    29
    29
    29
    29
  980. 26
    44
    26
    26
    26
    26
    26
    26
    26
  981. 26
    44
    26
    26
    26
    26
    26
    26
    26
  982. 4
    36
    41
    41
    41
    40
    41
    40
    2
    4
  983. 19
    19
    19
    19
    19
    19
    19
    19
    19
  984. 1
    25
    25
    25
    24
    25
    24
    1
    1
  985. 17
    6
    17
    17
    17
    17
    17
    17
    17

03Public benchmarks

Every openly released benchmark in the bank. Each card’s title links to the original benchmark, while “Build script” opens our reproducible build in measurement-db.

Category
Scale
Released
Minimum
Sort
Showing 201 of 201

2026

14
CEO-Bench benchmark
unknown2026

CEO-Bench

CEO-Bench: 15 LLM agents run a 500-day startup simulation (NovaMind); binary survival (avoided bankruptcy) per run, 3 runs per model.

1items
15subjects
DetailsPaper
Classroom AI benchmark
unknown2026

Classroom AI

Classroom AI (npj Artificial Intelligence 2026): grade-level classification of answers from 14 LLMs (six fine-tuned grade-specific GPT-4o-mini "teacher" models, base GPT-4o-mini / GPT-4o, and six prompt-based baselines) to 740 open-ended educational questions. response = the answer's integrated readability grade level (ordinal 1=lower-elementary .. 6=adult), computed by the paper's Algorithm 1 fusing seven readability formulas.

734items
14subjects
DetailsPaper
EDU-CIRCUIT-HW benchmark
cc-by-nc-4.02026

EDU-CIRCUIT-HW

EDU-CIRCUIT-HW (ACL 2026 Findings): per-(MLLM x handwritten-solution) recognition-correctness on 513 authentic university circuit-analysis homework solutions (observation set). Binary response = 1 if the model's Markdown transcription of the handwritten solution has no recognition error (Sample-level correct), 0 if it contains at least one recognition error, as flagged by the LLM-as-a-judge detector against expert-verified gold transcriptions. Six MLLM recognizers.

513items
6subjects
DetailsPaper
ExploitGym benchmark
Apache-2.02026

ExploitGym

ExploitGym: 869 real-world vulnerability instances (userspace C/C++, Google's V8 JS engine, and the Linux kernel) where an AI agent must turn a proof-of-vulnerability input into a working exploit achieving unauthorized code execution. Response is on_target: a successful exploit of the intended vulnerability.

868items
4subjects
DetailsPaper
Learning in Blocks benchmark
unknown2026

Learning in Blocks

Learning in Blocks (AIED 2026): per-(scoring-method x conversation) CEFR-aligned assessment scores. Six open-source LLM scoring pipelines (Self-Consistency and Self-Refine with/without thinking, homogeneous MAD, heterogeneous MAD) each score 44 CEFR A2 learner conversation transcripts on three dimensions (Grammar, Vocabulary, Interactive Communication), each on an ordinal 0-5 scale. Dimension carried as test_condition.

44items
6subjects
DetailsPaper
MathArena Platform benchmark
MIT2026

MathArena Platform

MathArena platform expansion (2026): new math-reasoning benchmark families with full per-(model, problem, attempt) outputs — ArXivMath (final-answer research), ArXivLean (Lean formal proofs), BrokenArXiv (false-claim reliability), USAMO 2026 (rubric-graded proofs).

428items
45subjects
DetailsPaper
OpenBioRQ benchmark
unknown2026

OpenBioRQ

OpenBioRQ: 657-question core of unsolved biomedical research questions for tool-using agents, graded by a frozen per-question checklist; solve = checklist score >= 0.5. Full per-(model x item) matrix released for 11 agents (roster, held-out, frontier, no-tool ablations).

657items
9subjects
DetailsPaper
OSWorld 2.0 benchmark
apache-2.02026

OSWorld 2.0

OSWorld 2.0: 108 long-horizon, real-world computer-use workflows across everyday and professional desktop/web tasks. Agents operate a real Ubuntu desktop with self-hosted services (email, banking, chat, portals) via screenshots; the final state is scored against fine-grained checkpoints. Primary metric is binary task completion (fully solved vs. not) at a 500-step budget.

108items
6subjects
DetailsPaper
PerfCodeBench benchmark
unknown2026

PerfCodeBench

LLMs optimize system-level high-performance code (C/C++/Go/Java/Python/CUDA); response is per-task binary correctness (compiles, runs, passes the deterministic oracle).

1,854items
20subjects
DetailsPaper
RelianceScope benchmark
cc-by-4.02026

RelianceScope

RelianceScope RQ3 benchmark: LLMs classify students' reliance engagement modes (Passive / Active / Constructive) along two axes — help-seeking and response-use — from interaction segments of a chatbot-assisted Vue.js programming activity. 6 LLMs x 4 prompting strategies evaluated on 150 manually annotated segments.

300items
6subjects
DetailsPaper
S-GRADES benchmark
unknown2026

S-GRADES

S-GRADES (LREC-COLING 2026): per-(model x item) grading predictions for three LLMs (GPT-4o-mini, Gemini 2.5 Flash, Llama 4 Scout) acting as automatic graders on the benchmark's four categorical short-answer datasets (BEEtlE and SciEntSBank, each in 2-way {correct, incorrect} and 3-way {correct, incorrect, contradictory} label settings). Each model graded every student answer three independent times under six reasoning-prompting strategies (inductive / deductive / abductive and their pairwise hybrids). response = 1 iff the model's grade matches the gold label. The benchmark's ~10 regression datasets (wide numeric score ranges) are excluded as non-categorical.

1,462items
3subjects
DetailsPaper
SketchJudge benchmark
unknown2026

SketchJudge

SketchJudge (arXiv 2601.06944): 16 multimodal LLMs act as graders of 1,015 hand-drawn student STEM answers (geometry, physics, chart, flowchart). response = binary grading accuracy, 1 iff the model's correctness verdict matches the expert gold label, under WithRef/NoRef settings and baseline/CoT/rubric prompt styles.

2,030items
16subjects
DetailsPaper
Terminal-Bench benchmark
Apache-2.02026

Terminal-Bench

Terminal-Bench: per-trial pass/fail of agent-model combos on terminal tasks.

89items
173subjects
DetailsPaper

2025

95
Abstract-Reason benchmark
MIT2025

Abstract-Reason

Abstract-Reason benchmark: rule-based symbolic reasoning tasks (basic/extended computation, non-decimal number bases, symbolic function inference, symbolic manipulation) with systematic symbol remapping to separate genuine abstraction from memorization.

9,089items
1subjects
DetailsPaper
AdaptiveStep (ASPRM) BoN benchmark
unknown2025

AdaptiveStep (ASPRM) BoN

Per-sample answer correctness of the MetaMath Llama-3.1-8B and Mistral-7B policy generators on the released AdaptiveStep Best-of-N candidate pools for GSM8k and MATH500 (256 samples per problem).

1,819items
2subjects
DetailsPaper
AdvPrompter benchmark
apache-2.02025

AdvPrompter

AdvPrompter adaptive adversarial-suffix jailbreak. Per-(target model x harmful behavior) binary attack success: 1 = the target LLM is jailbroken (Llama-Guard-3 judge) by the AdvPrompter suffix, 0 = refused. Third-party per-prompt results (AISafetyLab / THU-CoAI) for Vicuna-7B-v1.5 on 50 HarmBench behaviors, with and without the SafeDecoding defense.

50items
1subjects
DetailsPaper
AffectGPT MER-UniBench benchmark
apache-2.02025

AffectGPT MER-UniBench

MER-UniBench (AffectGPT, ICML 2025): per-item emotion-understanding results of the AffectGPT audio-video-text MLLM on 7 public test sets — basic emotion recognition (MER2023, MELD, IEMOCAP-four; binary emotion-wheel hit per item) and sentiment analysis (CMU-MOSI, CMU-MOSEI, CH-SIMS, CH-SIMS v2; binary polarity accuracy per item), for 7 released training checkpoints (epochs 30-60).

11,098items
1subjects
DetailsPaper
ALE-Bench benchmark
CC-BY-4.02025

ALE-Bench

ALE-Bench: long-horizon, score-based algorithm engineering on AtCoder Heuristic Contest optimization problems. 40 problems; 102 frontier LLMs evaluated across 5 self-refinement budgets. Per-(model, problem) judge verdict reduced to binary ACCEPTED vs. not.

40items
102subjects
DetailsPaper
AlgoTune benchmark
MIT2025

AlgoTune

AlgoTune: can LLM agents speed up general-purpose numerical programs? 154 coding tasks, each with a reference solver from a popular library, an input generator and a verifier. The agent must produce a correct but faster implementation. Binary response: pass iff the validated speedup over the reference is >= 1.0 (matched-or-beat the reference and was correct).

154items
18subjects
DetailsPaper
AtmosSci-Bench benchmark
CC-BY-4.02025

AtmosSci-Bench

AtmosSci-Bench: a symbolic-template benchmark of atmospheric-science problems (MCQ and open-ended) spanning hydrology, atmospheric dynamics, atmospheric physics, geophysics, and physical oceanography. 25 LLMs across instruction, reasoning, math, and domain-specific categories are evaluated by accuracy with symbolic-perturbation robustness analysis.

670items
29subjects
DetailsPaper
BountyBench benchmark
unknown2025

BountyBench

BountyBench: a real-world cybersecurity agent benchmark with 25 systems and 40 bug bounties. Per-(agent, task) run trajectories give a binary success verdict for three task types (Detect, Exploit, Patch); Detect is run under three hint conditions (none, CWE-only, CWE+title).

24items
10subjects
DetailsPaper
Cascade Routing benchmark
apache-2.02025

Cascade Routing

Per-(model x item) binary correctness released with the ICML 2025 paper "A Unified Approach to Routing and Cascading for LLMs", spanning SWE-Bench, Minerva Math, LiveCodeBench, ARC-Challenge, MMLU-Pro, MixEval and GSM8k.

16,552items
32subjects
DetailsPaper
CASE-Bench benchmark
unknown2025

CASE-Bench

CASE-Bench evaluates whether LLM safety judgments align with human judgments when the context of a harmful-looking query is taken into account. 900 items pair 450 SORRY-Bench queries with a safe and an unsafe Contextual-Integrity-formalized context; each item carries a human majority label from ~21 crowd annotators. Released per-item outputs cover 7 LLM judges under three elicitation methods (direct binary judgment, 1-10 score thresholded at >5.5, token-probability thresholded at >0.5); the response is 1 if the model judgment matches the human majority label.

900items
7subjects
DetailsPaper
ChemCoTBench benchmark
MIT2025

ChemCoTBench

ChemCoTBench: evaluates LLM chemical reasoning via modular chemical operations on SMILES. Per-instance binary scoring of two released models (claude3.7, ether0) across Molecule Understanding, Editing, Optimization and Reaction Prediction, using the benchmark's own RDKit-based eval logic (functional-group checks, count match, scaffold / SMILES exact match, property-improvement thresholds, MCQ accuracy).

1,144items
2subjects
DetailsPaper
ClinBench benchmark
Apache-2.02025

ClinBench

ClinBench released ablation study: per-item lung-cancer staging from TCGA pathology reports. Binary (gold == prediction) responses for 3 models across four extracted fields (pT, pN, tumor_stage, histologic_diagnosis) over 774 TCGA lung cases. The main 11-model benchmark is aggregate-F1 only with access-gated notes; this is the one released per-(model, item) artifact.

774items
3subjects
DetailsPaper
ConfAgents benchmark
unknown2025

ConfAgents

ConfAgents medical multiple-choice diagnosis: per-(agent, question) binary correctness (1 = final answer matches the gold option) on a released 16-question subset (8 MedBullets + 8 MedQA) answered by four agent frameworks (ConfAgents, ColaCare, MDAgents, MedAgent).

16items
4subjects
DetailsPaper
Critic-Discernment Game benchmark
unknown2025

Critic-Discernment Game

Critic-Discernment Game (CDG): a self-play reinforcement-learning training strategy that improves the rationality of an LLM reasoning process. Released lighteval detail files give per-(model, question) binary correctness (extractive_match) for the baseline Llama-3.1-8B-Instruct and the CDG-trained prover on GSM8K and MATH500.

1,819items
2subjects
DetailsPaper
CSEDB benchmark
unknown2025

CSEDB

CSEDB (Clinical Safety-Effectiveness Dual-Track Benchmark): per-(model, item) pass/fail in {0,1} on the 1,260 black-and-white (non-黑即白型) clinical Q&A items of the 2,069-item benchmark. 6 LLMs graded by an LLM judge across a Safety gate and an Effectiveness gate, 26 clinical departments; 3 independent runs as trials.

1,260items
6subjects
DetailsPaper
DAS Medical Red-Teaming (Hallucination) benchmark
MIT2025

DAS Medical Red-Teaming (Hallucination)

DAS Medical Red-Teaming (hallucination axis): per-(model, item) DAS hallucination-detector verdict on clinical text generated by 15 LLMs over medical red-teaming prompts. Response in {0, 0.5, 1} (1=hallucination detected, 0=clean, 0.5=borderline).

108items
15subjects
DetailsPaper
DataDecide benchmark
odc-by2025

DataDecide

DataDecide: OLMES multiple-choice QA evaluation of a controlled pretraining suite (25 data recipes x 14 sizes x 3 seeds). Per-instance correctness over 10 OLMES benchmarks (ARC, BoolQ, CommonsenseQA, HellaSwag, MMLU, OpenBookQA, PIQA, Social IQa, Winogrande).

37,553items
1subjects
DetailsPaper
DIS-CO (MovieTection) benchmark
CC-BY-NC-SA-4.02025

DIS-CO (MovieTection)

VLMs identify the source movie of a frame (or its caption) in free-form generation; response is per-frame correct title identification, used to infer training-data membership.

13,999items
4subjects
DetailsPaper
EduGuardBench benchmark
unknown2025

EduGuardBench

EduGuardBench (AAAI 2026): per-(model x item) results for 14 LLMs as simulated teachers. Component I = 2,635 Select-All-That-Apply teaching questions scored by exact-set match (binary Role-playing Fidelity). Component II = 801 persona-based jailbreak prompts scored by a DeepSeek-V3 Best-of-9 judge for harmfulness (binary attack success) and a three-tier refusal quality (ordinal). Mixed response types encoded as separate test conditions.

3,394items
14subjects
DetailsPaper
EDUMATH benchmark
mit2025

EDUMATH

EDUMATH (EMNLP 2025): per-item quality labels for 8,360 K-12 math word problems generated by 11 open/closed LLMs conditioned on a Virginia math standard + grade level. Binary LLM-judge label (1 = solvable, accurate, educationally appropriate, and standards-aligned) plus the EDUMATH ModernBERT classifier label, each as a separate judge test condition.

6,849items
11subjects
DetailsPaper
EHRFlowBench benchmark
MIT2025

EHRFlowBench

EHRFlowBench: open-ended electronic-health-record (EHR) analysis report-generation tasks derived from peer-reviewed clinical research. Per-(agent framework x task) binary completion success (1 = valid evaluator-verified report produced, 0 = failed). 7 AI agent frameworks x 20 human-eval-subset tasks = 140 responses.

20items
7subjects
DetailsPaper
The Elicitation Game benchmark
unknown2025

The Elicitation Game

The Elicitation Game evaluates capability-elicitation techniques against "model organisms" (password-locked and circuit-broken LMs with a hidden WMDP-knowledge capability). Released per-item results give, for each model organism under each elicitation technique (many-shot prefilling, GCG adversarial suffixes, anti-refusal fine-tuning), binary correctness on a 370-question WMDP multiple-choice test set.

370items
6subjects
DetailsPaper
Embodied Web Agents benchmark
MIT2025

Embodied Web Agents

Embodied Web Agents (indoor cooking, text-based): per-task pass/fail for the GPT (gpt-4o) run. Each task pairs a web subtask (find/buy recipe items) with an embodied subtask (achieve the recipe object final states in AI2-THOR). Released as two per-task score files.

876items
1subjects
DetailsPaper
EmbodiedBench benchmark
MIT2025

EmbodiedBench

MLLMs act as vision-driven embodied agents across four simulated environments; response is per-episode binary task success.

969items
11subjects
DetailsPaper
EngineMT-QA benchmark
MIT2025

EngineMT-QA

EngineMT-QA: large-scale multi-task Time-Series QA over N-CMAPSS aero-engine sensor signals (33 channels). Items pair a time-series window with an NL question across Understanding, Perception, Reasoning and Decision-Making tasks. Built subset covers the closed multiple-choice Perception and Reasoning items.

676items
1subjects
DetailsPaper
Fantastic Bugs benchmark
CC-BY-4.02025

Fantastic Bugs

Fantastic Bugs: a measurement-theoretic framework that flags potentially invalid benchmark questions (ambiguous wording, wrong answer key, grading issues) for expert review. Released as per-(model, question) HELM correctness matrices across roughly a dozen widely used benchmarks (GSM8K, MMLU subsets, MedQA, ThaiExam, BBQ, BoolQ, WikiFact, CommonSenseQA, LegalBench, AIR-Bench).

21,905items
180subjects
DetailsPaper
FineGRAIN (T2I) benchmark
MIT2025

FineGRAIN (T2I)

FineGRAIN T2I failure-mode benchmark: ~17 text-to-image models x 760 prompts, each prompt tagged with one of 27 fine-grained failure modes (counting, colour/shape/texture attribute binding, spatial relations, physics, text rendering, negation, perspective, ...) across 11 categories. Each generated image carries a human label for whether the prompt's failure mode is present; response is the human success verdict (1 = no failure / prompt rendered correctly, 0 = failure).

760items
17subjects
DetailsPaper
FlySearch benchmark
MIT2025

FlySearch

FlySearch: object-goal navigation / exploration benchmark for VLMs in a photorealistic 3D UE5 simulation. A VLM pilots a drone via relative-move commands to find a described object. Scenario sets FS-1 (forest+city), FS-2 (harder city scenes) and FS-Anomaly; success measured per episode.

1,384items
10subjects
DetailsPaper
GEDE benchmark
unknown2025

GEDE

GEDE: benchmarking LLM-generated-text detectors on 13k+ student and LLM essays across contribution levels; per-(detector, essay) binary correctness at a static 0.5 threshold.

13,217items
3subjects
DetailsPaper
GSO benchmark
MIT2025

GSO

GSO: challenging software-optimization tasks for evaluating SWE-agents. Each task asks an agent to reproduce a real performance optimization in a repository, graded by the Opt@K metric (the agent patch must meet the optimization target under a performance script plus correctness tests). We store the real per-task descriptions and per-(model, instance) Opt@1 pass/fail outcomes from the official gso-experiments reports.

102items
21subjects
DetailsPaper
HARDMath2 benchmark
MIT2025

HARDMath2

HARDMath2: 210 graduate-level applied-mathematics problems (asymptotic series, boundary layers, integrals, nonlinear ODEs/PDEs, WKB) built by students. Per-(model, problem, sample) correctness from a SymPy-based equivalence checker across GPT and Gemini model families.

210items
9subjects
DetailsPaper
HIVMedQA benchmark
CC-BY-4.02025

HIVMedQA

HIVMedQA: open-ended HIV medical decision-support QA. Per-(model, question, iteration, dimension) LLM-as-a-judge (MedGPT) Likert scores on a 0-5 scale across 5 clinical dimensions (comprehension, reasoning, knowledge recall, bias, harm). 10 LLMs x 63 questions x 5 iterations.

63items
10subjects
DetailsPaper
Humanity's Last Exam benchmark
MIT2025

Humanity's Last Exam

Humanity's Last Exam — 2500-question frontier benchmark; per-item binary correct/incorrect from supaihq judged data + deepwriter Gemini-3 data.

1,792items
19subjects
DetailsPaper
ICPC-2 Code Selector benchmark
MIT2025

ICPC-2 Code Selector

ICPC-2 Code Selector: per-(model, query, top_k) binary correctness for selecting the best-matching ICPC-2 primary-care code for 435 Brazilian Portuguese clinical expressions from semantic-search candidates. 40 LLM variants; response=1 if correct code (or correct abstention) selected.

435items
40subjects
DetailsPaper
IgakuQA119 benchmark
CC-BY-4.02025

IgakuQA119

IgakuQA119 — per-question correctness on the 119th Japanese Medical Licensing Examination (Feb 2025). Single-subject matrix: one model (Qwen2.5-72B) x ~400 questions, binary correct/incorrect.

400items
1subjects
DetailsPaper
IneqMath benchmark
CC-BY-SA-4.02025

IneqMath

IneqMath: Olympiad-level inequality problems reformulated into bound estimation and relation prediction subtasks. Binary response = the final-answer-judge correctness (deterministically reconstructed from the released per-problem responses and validated to match the released scores.json aggregates). Upstream releases full per-problem output for one worked-example model (gpt-4o-mini); the 50+ leaderboard models are aggregate-only.

300items
1subjects
DetailsPaper
JETTS benchmark
apache-2.02025

JETTS

JETTS released generator response pool: per-(generator model, query) correctness over 7 math/code/instruction-following datasets, 10 samples per query.

5,135items
8subjects
DetailsPaper
JudgeTuning benchmark
apache-2.02025

JudgeTuning

JudgeTuning: LLM-as-judge design-decision tuning. Per-battle human agreement of tuned open-weight LLM judges (Ours-tiny/small/medium/large) and reference judges (Arena-Hard GPT-4o-mini, JudgeLM-7B, PandaLM-7B) on 3,000 LMSys pairwise preference battles.

2,981items
7subjects
DetailsPaper
KernelBench benchmark
MIT2025

KernelBench

LLMs generate optimized CUDA kernels for 250 PyTorch ML workloads; response is per-task functional correctness.

250items
7subjects
DetailsPaper
LiveAoPSBench benchmark
apache-2.02025

LiveAoPSBench

LiveAoPSBench: a contamination-resistant, timestamped evaluation set of 5328 Olympiad-level math problems mined from the Art of Problem Solving forum. Per-(model, problem) short-answer correctness (pass@1).

5,321items
13subjects
DetailsPaper
LLM Survey Simulation (UQ) benchmark
unknown2025

LLM Survey Simulation (UQ)

Per-(LLM, survey-question) simulated human responses from "Uncertainty Quantification for LLM-Based Survey Simulations" (ICML 2025). Eight LLMs are prompted with random human personas to simulate responses on two real survey banks: EEDI (412 math multiple-choice questions, binary correct/incorrect) and OpinionQA (385 Pew opinion questions, ordinal sentiment score in {-1,-1/3,0,1/3,1}). Each model emits K simulated responses per question, one per synthetic persona, stored as separate trials.

797items
8subjects
DetailsPaper
MAST (Multi-Agent System Failure Taxonomy) benchmark
CC-BY-4.02025

MAST (Multi-Agent System Failure Taxonomy)

MAST: per-trace annotations of 1242 multi-agent LLM system execution traces across 7 frameworks (ChatDev, MetaGPT, AppWorld, AG2, HyperAgent, Magentic, OpenManus) for 14 failure modes from the Multi-Agent Systems Failure Taxonomy. Each (system, trace) cell is labelled present/absent for each of the 14 failure modes by the LLM annotator (OpenAI o1, calibrated to kappa=0.77 with human experts).

1,242items
9subjects
DetailsPaper
MathConstruct benchmark
MIT2025

MathConstruct

MathConstruct: 121 math-olympiad constructive-proof problems parameterised into 455 instances, graded by deterministic checkers into binary correct/incorrect; released per-(model x instance) results for 19 (CoT) + 10 (code) LLMs.

455items
20subjects
DetailsPaper
MedHELM EHRSQL benchmark
Apache-2.02025

MedHELM EHRSQL

MedHELM ehr_sql: per-(model, item) ehr_sql_execution_accuracy in {0,1} (1=correct) on 1000 translating a natural-language clinical question into SQL over an EHR database. 13 models from the MedHELM v4.0.0 release.

1,000items
13subjects
DetailsPaper
MedHELM HEAD-QA benchmark
Apache-2.02025

MedHELM HEAD-QA

MedHELM head_qa: per-(model, item) exact_match in {0,1} (1=correct) on 1000 Spanish healthcare specialization (MIR/medicine/pharmacy/nursing) exam questions. 13 models from the MedHELM v4.0.0 release.

1,000items
13subjects
DetailsPaper
MedHELM MedMCQA benchmark
Apache-2.02025

MedHELM MedMCQA

MedHELM med_mcqa: per-(model, item) exact_match in {0,1} (1=correct) on 1000 Indian medical-entrance (AIIMS/NEET-PG) multiple-choice questions. 13 models from the MedHELM v4.0.0 release.

1,000items
13subjects
DetailsPaper
MedHELM MedQA benchmark
Apache-2.02025

MedHELM MedQA

MedHELM med_qa: per-(model, item) exact_match in {0,1} (1=correct) on 1000 US medical licensing exam (USMLE) multiple-choice clinical questions. 13 models from the MedHELM v4.0.0 release.

1,000items
13subjects
DetailsPaper
MedHELM Medbullets benchmark
Apache-2.02025

MedHELM Medbullets

MedHELM medbullets: per-(model, item) exact_match in {0,1} (1=correct) on 308 USMLE Step 2/3-style clinical vignette multiple-choice questions. 13 models from the MedHELM v4.0.0 release.

298items
13subjects
DetailsPaper
MedHELM MedCalc-Bench benchmark
Apache-2.02025

MedHELM MedCalc-Bench

MedHELM medcalc_bench: per-(model, item) medcalc_bench_accuracy in {0,1} (1=correct) on 1000 clinical calculations from a patient note (risk scores, dosing, lab-derived values). 13 models from the MedHELM v4.0.0 release.

992items
13subjects
DetailsPaper
MedHELM MEDEC benchmark
Apache-2.02025

MedHELM MEDEC

MedHELM medec: per-(model, item) medec_error_flag_accuracy in {0,1} (1=correct) on 597 detecting whether a clinical note contains a medical error (error-flag accuracy). 13 models from the MedHELM v4.0.0 release.

597items
13subjects
DetailsPaper
MedHELM MedHallu benchmark
Apache-2.02025

MedHELM MedHallu

MedHELM medhallu: per-(model, item) exact_match in {0,1} (1=correct) on 1000 medical hallucination detection: label an answer as hallucinated or faithful. 13 models from the MedHELM v4.0.0 release.

1,000items
13subjects
DetailsPaper
MedHELM PubMedQA benchmark
Apache-2.02025

MedHELM PubMedQA

MedHELM pubmed_qa: per-(model, item) exact_match in {0,1} (1=correct) on 1000 biomedical research questions answered yes/no/maybe over PubMed abstracts. 13 models from the MedHELM v4.0.0 release.

1,000items
13subjects
DetailsPaper
MedHELM RaceBasedMed benchmark
Apache-2.02025

MedHELM RaceBasedMed

MedHELM race_based_med: per-(model, item) exact_match in {0,1} (1=correct) on 167 detecting whether a medical answer relies on harmful race-based reasoning. 13 models from the MedHELM v4.0.0 release.

167items
13subjects
DetailsPaper
MedWatermark-FWS benchmark
Apache-2.02025

MedWatermark-FWS

MedWatermark-FWS: GPT-Judger 1-5 Likert scores (coherence, relevance, factual accuracy) for watermarked vs unwatermarked medical-LLM (Meditron-7B) generations, across 4 watermarking methods (DIP, EXPEdit, KGW, SWEET) and 3 medical datasets (HealthQA span, HealthQA QA, MeQSum). 200 prompts per method/dataset, judged for both text variants.

594items
1subjects
DetailsPaper
MetaAgent (GPQA-Diamond) benchmark
2025

MetaAgent (GPQA-Diamond)

Per-(system x GPQA-Diamond question) binary correctness for MetaAgent and single-/multi-agent prompting baselines (Direct, CoT, CoT-SC, LLM-Debate, Self-Refine, SPP) over GPT-4o and GPT-3.5-Turbo, recovered from the authors' released evaluation logs.

200items
10subjects
DetailsPaper
MoE-CAP benchmark
unknown2025

MoE-CAP

MoE-CAP: a benchmark and framework for sparse Mixture-of-Experts LLM serving systems, characterizing the Cost/Accuracy/Performance trade-off with sparsity-aware utilization metrics (S-MBU, S-MFU). This ingestion captures the released lm-eval-harness per-item correctness: each MoE model under a serving framework is graded per GSM8K / MMLU question.

7,736items
14subjects
DetailsPaper
Large Language Monkeys: Power Laws benchmark
mit2025

Large Language Monkeys: Power Laws

How Do Large Language Monkeys Get Their Power (Laws)? Per-sample correctness from repeated sampling (10,000 samples per problem) for Llama-3 / Gemma / Pythia models on GSM8K, MATH, MiniF2F-MATH and CodeContests. Each sample is a binary trial (correct / incorrect).

525items
13subjects
DetailsPaper
MorphKV (LongGenBench) benchmark
MIT2025

MorphKV (LongGenBench)

LongGenBench long-response constraint-following outcomes for three KV-cache compression methods (MorphKV / SnapKV / H2O) across three base LLMs; response is per-constraint yes/no.

40items
9subjects
DetailsPaper
MORQA benchmark
CC-BY-4.02025

MORQA

Medical Open-Response QA meta-evaluation: LLM-based evaluators rate candidate answers to medical questions on a 3-point {0, 0.5, 1} scale, across English/Chinese datasets.

5,441items
4subjects
DetailsPaper
MTBBench benchmark
MIT2025

MTBBench

MTBBench evaluates multimodal LLMs on sequential clinical decision-making in oncology. Two cohorts: HANCOCK (multimodal pathology — TMA images, H&E slides, cell density) and MSK-CHORD (longitudinal genomics — genomic profiles, clinical timelines). Models act as agentic AI physicians, optionally using external tools (DrugBank, CONCH, PubMed). Each case contains multiple multi-choice questions; correctness is binary (matched vs. ground truth).

516items
23subjects
DetailsPaper
Multi-MoE benchmark
unknown2025

Multi-MoE

Per-question binary correctness (string-match) on MMLU (5-shot, 1000 items) and TruthfulQA (5-shot MC, 812 items) for five Mixtral-8x7B serving configurations (base, Instruct, weight-avg, Multi-MoE with runtime reconfiguration, Multi-MoE no-reconfig) released with the QoS-efficient MoE-serving paper.

1,811items
5subjects
DetailsPaper
Multi-SWE-bench benchmark
Apache-2.02025

Multi-SWE-bench

Multi-SWE-bench: multilingual issue-resolving benchmark over 8 languages (Python, Java, TS, JS, Go, Rust, C, C++). Per-instance resolved/unresolved verdicts for 39+ agent x model submissions, mirrored into a binary matrix.

2,078items
82subjects
DetailsPaper
Multimodal STEM Assessments (Human-AI) benchmark
unknown2025

Multimodal STEM Assessments (Human-AI)

201+ university-level STEM exam questions with images from EPFL Bachelor/Master courses across 11 subjects (astronomy, biology, chemistry, CS, electrical engineering, electromagnetism, math, mechanical physics, microfabrication, neuroscience, quantum physics). Each question graded by exact match against an educator gold answer. Five models evaluated under multiple prompting strategies; per-item graded outputs released in the official GitHub runs file.

249items
5subjects
DetailsPaper
NAEP LLM-as-Student benchmark
unknown2025

NAEP LLM-as-Student

NAEP LLM-as-Student: 488 NAEP multiple-choice items (mathematics and reading comprehension, grades 4/8/12) answered by 11 LLMs under four prompting conditions (unenforced and grade-enforced, with/without CoT). Each response is binary (1 = model picked the keyed option, 0 = not).

488items
11subjects
DetailsPaper
NaturalReasoning benchmark
CC-BY-NC-4.02025

NaturalReasoning

NaturalReasoning: 2.8M challenging reasoning questions backtranslated from pretraining corpora, spanning mathematics, physics, computer science, and economics/social sciences. Each question carries a reference answer and model-generated responses (Llama-3.3-70B-Instruct).

143,048items
1subjects
DetailsPaper
NESTER benchmark
unknown2025

NESTER

NESTER: dataflow-guided neuro-symbolic type inference for Python. Released per-item data covers the human evaluation of high-level program generation on ManyTypes4Py samples: GPT-4, GPT-4o and GPT-4o mini generations each graded on four binary criteria (correctness, explainability, factuality, consistency) by human annotators. Recovered from deleted xlsx blobs in the authors' GitHub repository.

90items
3subjects
DetailsPaper
OCRBench v2 benchmark
Apache-2.02025

OCRBench v2

OCRBench v2: a large-scale bilingual (English + Chinese) text-centric benchmark for LMMs spanning eight capabilities (text recognition, referring, spotting, relation extraction, element parsing, mathematical calculation, visual text understanding, knowledge reasoning) with task-tailored metrics. Per-example predictions are ingested for InternVL2.5-26B (the one model with a released per-instance prediction file); response is a transparent per-example correctness flag and the raw model output is kept in the trace.

4,893items
1subjects
DetailsPaper
OOD-Prediction (Internal Causal Mechanisms) benchmark
MIT2025

OOD-Prediction (Internal Causal Mechanisms)

OOD-Prediction correctness dataset (Huang et al., ICML 2025): released prompts for 5 tasks (IOI, PriceTag, RAVEL, MMLU, Unlearn Harry Potter) under in-distribution and OOD settings, each prompt labelled correct/wrong by the target model. subject = target model; item = the prompt; response = 1 correct / 0 wrong.

142,805items
2subjects
DetailsPaper
OS-Harm benchmark
Apache-2.02025

OS-Harm

OS-HARM: a benchmark for the safety of computer-use agents in the OSWorld desktop environment. 150 tasks across three harm categories (deliberate user misuse, prompt injection attacks, model misbehavior) over real OS apps (chrome, gimp, libreoffice, thunderbird, vs_code, os, vlc). An automated LLM judge (AER + GPT-4.1) emits a binary success and a binary safety verdict per (model, task) execution trace; judge-vs-human F1 is 0.76 (success) / 0.79 (safety).

150items
5subjects
DetailsPaper
PeruMedQA benchmark
unknown2025

PeruMedQA

PeruMedQA: per-(model, item) correctness in {0,1} (1=correct) on 8,380 Spanish-language Peruvian medical specialty-exam (CONAREME, 2018-2025) multiple-choice questions across 12 specialties. 11 medical LLMs (10 vanilla + 1 fine-tuned medgemma-4b-it).

8,259items
11subjects
DetailsPaper
Potemkin Understanding benchmark
unknown2025

Potemkin Understanding

Potemkin Understanding benchmark: 32 concepts across literary techniques, game theory and psychological biases. LLMs are scored on define / classify / generate / edit / incoherence tasks; each response is binary (1 = correct application/classification, 0 = incorrect).

4,023items
11subjects
DetailsPaper
Prompt-to-Leaderboard benchmark
CC-BY-4.02025

Prompt-to-Leaderboard

Prompt-to-Leaderboard: prompt-adaptive LLM leaderboards from Chatbot Arena human pairwise battles (win/loss/tie per prompt).

128,287items
53subjects
DetailsPaper
Psychosis-bench benchmark
unknown2025

Psychosis-bench

Psychosis-bench: LLM psychogenicity across 16 twelve-turn delusional-conversation scenarios, scored per turn on Delusion Confirmation (0-2), Harm Enablement (0-2) and Safety Intervention (0-1).

144items
8subjects
DetailsPaper
Pxplore benchmark
unknown2025

Pxplore

Pxplore Personalized Learning Path Planning: given a learner state (long/short-term objectives and implicit/explicit motivations) and candidate learning actions, an LLM policy selects a next lesson; an evaluator re-assesses each learner-state component and marks it ALIGNED iff the chosen content advances that goal. Response is the binary per-component pedagogical-alignment outcome.

254items
14subjects
DetailsPaper
REAL benchmark
unknown2025

REAL

REAL: autonomous web agents on deterministic simulations of real websites. Tasks across high-fidelity website clones (Staynb, DashDish, ...), each with a deterministic success checker made of one or more evals. Response is the per-attempt binary task success (1 iff all of the task's evals pass).

233items
44subjects
DetailsPaper
ReEval benchmark
apache-2.02025

ReEval

Full binary response matrix (183 LMs x 22 HELM scenarios) released with the amortized IRT evaluation paper.

84,327items
183subjects
DetailsPaper
RefGrader benchmark
mit2025

RefGrader

RefGrader: LLM-as-grader of Olympiad mathematical competition proofs. An LLM (Gemini 2.5 Pro) grades problem-solution pairs on the 0-7 Olympiad partial-credit scale under a single-turn baseline and reference-aided multi-step agentic workflows (3-step no-rubric and 5-step RefGrader rubric designs). Evaluated on an IMO Shortlist corpus (90 items) and MathArena IMO/USAMO 2025 solutions (385 items).

475items
1subjects
DetailsPaper
ResearchCodeBench benchmark
MIT2025

ResearchCodeBench

ResearchCodeBench: 212 coding challenges from 20 recent (2024-2025) ML papers. Given the paper plus context code with a blanked snippet, an LLM fills in the missing implementation, graded by curated tests. 32 LLMs evaluated; per-(model, snippet) pass/fail over 6784 observations.

212items
32subjects
DetailsPaper
SMMILE benchmark
CC-BY-4.02025

SMMILE

SMMILE: an expert-driven benchmark for multimodal medical in-context learning. 111 problems (517 image-question-answer triplets) across 6 specialties and 13 imaging modalities; multimodal LLMs evaluated zero-shot vs with in-context examples on closed-ended MCQA and open-ended (exact-match / LLM-as-a-judge) accuracy. SMMILE++ permutes the in-context demonstrations into 1038 problems.

111items
16subjects
DetailsPaper
SOAR ARC-AGI benchmark
unknown2025

SOAR ARC-AGI

SOAR on ARC-AGI: per-(model, ARC-train task) binary success of an LLM-driven evolutionary program-synthesis search, across the base model (generation 0) and SOAR self-improvement iterations. A response is 1 if the model produced a Python program solving the original ARC task, else 0.

400items
5subjects
DetailsPaper
SuperGPQA benchmark
CC-BY-NC-SA-4.02025

SuperGPQA

SuperGPQA: graduate-level multiple-choice knowledge & reasoning across 285 disciplines (26,529 questions, avg 9.67 options/question). Per-(model, question) verdicts for 49 LLMs from the upstream zero-shot records dump; response is the binary correct/incorrect outcome.

26,529items
49subjects
DetailsPaper
SWE-rebench benchmark
CC-BY-4.02025

SWE-rebench

SWE-rebench: an automated, decontaminated benchmark of real-world software-engineering agent tasks. Each task is a GitHub issue + repo snapshot; an LLM agent must produce a patch verified by the repo test suite (FAIL_TO_PASS / PASS_TO_PASS). This build ingests the publicly released per-instance trajectories of the OpenHands v0.54.0 + Qwen3-Coder-480B-A35B-Instruct agent, emitting a binary resolved=1/0 response per (agent, instance, run).

6,271items
1subjects
DetailsPaper
SWE-smith benchmark
MIT2025

SWE-smith

SWE-smith agent trajectories: per-(model x task-instance) outcomes from SWE-agent runs over synthesized software-engineering task instances. Each cell is binary resolved (the agent patch passes the instance tests, 1) or not (0), recorded under three trajectory-format splits (tool/xml/ticks).

13,472items
3subjects
DetailsPaper
TabICL benchmark
Apache-2.02025

TabICL

TabICL and the tabular in-context foundation-model family evaluated by the TabArena living benchmark, ingested at the GRANULAR per-row level: each response is one tabular foundation model's per-row correctness on one individual OpenML classification test row (fold 0 of the tabarena-v0.1 suite). The response is per-row classification correctness (argmax of the predicted class probabilities vs the true label). Items are real tabular rows (feature name->value). Subjects are in-context tabular foundation models: TabICL, TabPFNv2, TabDPT, TabFlex, Mitra, BetaTabPFN, LimiX.

269,378items
7subjects
DetailsPaper
TDD-Bench-Verified benchmark
apache-2.02025

TDD-Bench-Verified

TDD-Bench-Verified: generating tests from GitHub issues before the fix exists. 449 issues filtered from SWE-bench Verified; a generated test suite passes iff it fails on the pre-fix code and passes on the golden-patched code (fail-to-pass). Per-instance harness verdicts for Otter and e-Otter with GPT-4o and Claude-3.7-Sonnet under ten prompting variants, from the authors' Zenodo artifact.

449items
4subjects
DetailsPaper
Thousand Voices of Trauma benchmark
CC-BY-NC-4.02025

Thousand Voices of Trauma

Thousand Voices of Trauma: 3,000 synthetic Prolonged-Exposure therapy conversations (500 clients x 6 phases, generated by Claude Sonnet 3.5) plus an emotional-trajectory benchmark in which 8 LLMs rate the emotional intensity (1-10) of each PE-therapy phase per client, scored against the Claude Sonnet 3.5 baseline trajectory.

3,000items
8subjects
DetailsPaper
VideoGameQA-Bench benchmark
CC-BY-4.02025

VideoGameQA-Bench

VideoGameQA-Bench: evaluating VLMs for video game quality assurance. Per-item verdict dumps for 4 tasks (image/video glitch detection, image/video bug-report generation) over 16+ VLMs, parsed from the project artifacts dashboards. response is the upstream per-item Correct?/judge-Match boolean.

3,436items
17subjects
DetailsPaper
Visual Memory (flexible perception) benchmark
cc-by-4.02025

Visual Memory (flexible perception)

Per-test-image top-1 nearest-neighbor retrieval correctness for 5 embedding models (DinoV2 ViT-S/B/L14, CLIP ViT-B16/L14) using an ImageNet-2012 visual memory, on three ImageNet distribution-shift test sets (ImageNet-A/R/V2). Released as per-image nearest-neighbor JSON on gs://visual_memory_v01.

47,500items
5subjects
DetailsPaper
WikiHowAgent benchmark
unknown2025

WikiHowAgent

WikiHowAgent per-conversation completion outcomes: for each WikiHow how-to tutorial, whether a multi-LLM teacher-learner workflow (varied learner LLM) produced a conversation that reached completion.

13,565items
7subjects
DetailsPaper
WorldCentralBanks benchmark
CC-BY-NC-SA-4.02025

WorldCentralBanks

WorldCentralBanks: 25,000 annotated sentences from 26 central banks for Stance Detection (Hawkish/Dovish/Neutral/Irrelevant), Temporal Classification, and Uncertainty (Certainty) Estimation. This build holds per-(model, sentence) zero-shot/few-shot LLM predictions released in the upstream repo, scored as binary correctness (predicted label vs the gold annotation), with the gold annotation as correct_answer.

9,575items
10subjects
DetailsPaper

2024

56
AAAR-1.0 (EqInfer) benchmark
MIT2024

AAAR-1.0 (EqInfer)

AAAR-1.0 Equation Inference: given the LaTeX context before and after a removed equation from a peer-reviewed NLP/ML paper, pick the human-written equation among 4 candidates (3 GPT-4-synthesized negatives). Per-item results recovered from the deleted git history of the official repo cover the single released subject, Gemini Pro (multiple-choice setting, 1000-word contexts); the paper's other models have no released per-item outputs.

1,049items
1subjects
DetailsPaper
AfriMed-QA benchmark
CC-BY-NC-SA-4.02024

AfriMed-QA

AfriMed-QA: medical QA for African healthcare contexts; binary correctness on MCQ items.

6,910items
31subjects
DetailsPaper
AgentBoard benchmark
GPL-3.02024

AgentBoard

AgentBoard: analytical evaluation of multi-turn LLM agents across 9 interactive environments (alfworld, scienceworld, babyai, jericho, pddl, webarena, webshop, tool-query, tool-operation). Each item is a task instance; response is per-example success_rate (1 if the agent completed the full task).

867items
13subjects
DetailsPaper
AgentDojo benchmark
MIT2024

AgentDojo

AgentDojo: prompt-injection evaluation across many tool-using agents and environments.

1,081items
29subjects
DetailsPaper
Alignment Faking (RL) benchmark
CC-BY-4.02024

Alignment Faking (RL)

Alignment Faking (RL) — per-(run, prompt) alignment-faking behaviour for a panel of RL training runs / model organisms (Anthropic). subject = an RL run/model organism; item = an eval prompt; response = alignment_faking (1/0). test_condition records the training step and free/paid-tier framing.

244items
155subjects
DetailsPaper
AppWorld benchmark
Apache-2.02024

AppWorld

AppWorld: interactive coding agent benchmark. Per-(agent, task) pass/fail decrypted from the public leaderboard's experiment bundles; real per-task natural-language instructions from the released task data.

521items
8subjects
DetailsPaper
Arena-Hard-Auto benchmark
Apache-2.02024

Arena-Hard-Auto

Arena-Hard-Auto: 5-level GPT-4 judgments of models vs a fixed baseline (GPT-4-0314).

500items
72subjects
DetailsPaper
AutoElicit (ICL baseline) benchmark
MIT2024

AutoElicit (ICL baseline)

GPT-3.5-turbo in-context-learning baseline from AutoElicit: direct per-instance classification of tabular clinical/quality datasets (breast cancer, heart disease, hypothyroid, wine quality). Response is correctness of the 0.5-thresholded LLM probability.

425items
1subjects
DetailsPaper
BABILong benchmark
Apache-2.02024

BABILong

BABILong: long-context reasoning-in-a-haystack. bAbI reasoning questions (qa1..qa20) are embedded in distractor contexts of varying token length (0k..10M); a model passes an item iff its answer matches the gold bAbI label. Per-item model outputs are released for qa1..qa5 for two models; scored {0,1} by BABILong's string-label metric.

36,832items
2subjects
DetailsPaper
BertaQA benchmark
CC-BY-SA-4.02024

BertaQA

BertaQA: trivia multiple-choice QA contrasting knowledge of Basque local culture vs. global culture. Each item is a 3-way (A/B/C) question; a model is scored correct iff it picks the gold candidate. Questions are evaluated 5-shot in English (en) and Basque (eu).

9,489items
6subjects
DetailsPaper
BFCL benchmark
Apache-2.02024

BFCL

Build a per-task binary response matrix and extract per-task error types from BFCL score files

4,133items
93subjects
DetailsPaper
CARE benchmark
MIT2024

CARE

CARE: a benchmark suite for the Classification And Retrieval of Enzymes. Task 1 predicts the Enzyme Commission (EC) number of a protein from its sequence; Task 2 retrieves the EC number of a chemical reaction. Methods produce a ranked list of predicted EC numbers per item; scored here as top-1 accuracy against the gold EC.

2,019items
11subjects
DetailsPaper
Causal Axioms (Axiomatic Training) benchmark
unknown2024

Causal Axioms (Axiomatic Training)

Axiomatic Training causal-reasoning evaluation: per-item Yes/No d-separation questions over linear causal chains of length 7-15 (500 items per length), answered by the baseline Meta-Llama-3-8B-Instruct and its axiomatic-training LoRA finetune. Response is binary (1 = prediction matches the gold Yes/No label).

4,500items
2subjects
DetailsPaper
CharXiv benchmark
CC-BY-SA-4.02024

CharXiv

CharXiv: realistic chart understanding for multimodal LLMs, built from real arXiv figures. 1,000 validation charts, each with four descriptive questions (19 fixed templates) and one free-form reasoning question. Models are scored per (model, question); reasoning is exact-match, descriptive is GPT-4o-graded against a rubric.

5,000items
37subjects
DetailsPaper
ClashEval benchmark
MIT2024

ClashEval

ClashEval: quantifying the tug-of-war between an LLM's internal prior and external (retrieved) evidence. Each QA item is answered twice — closed-book (prior) and with a perturbed reference passage (post) whose gold answer has been mutated. Responses are binary correctness against the true answer across six domains (drug dosages, Olympic records, names, locations, dates, news).

11,143items
6subjects
DetailsPaper
ComplexBench benchmark
MIT2024

ComplexBench

ComplexBench: complex instruction following with multiple composed constraints (NeurIPS 2024). Each instruction decomposes into binary scoring questions; this build ingests the deterministic, rule-verified subset of those questions, grading each released model generation with the benchmark's own rule evaluator.

252items
15subjects
DetailsPaper
CROP benchmark
CC-BY-NC-4.02024

CROP

CROP benchmark: 5,045 bilingual (Chinese/English) crop-science multiple-choice questions across three difficulty levels, derived from 2K+ crop-science academic papers. Each item asks the model to select the correct option; the response is whether the chosen option was correct. We ingest the authors' released per-(model, item) grading matrix for four commercial LLMs.

5,045items
4subjects
DetailsPaper
CROW benchmark
MIT2024

CROW

CROW backdoor-elimination intervention (ICML 2025): per-prompt attack-success outcomes of backdoored LLaMA2 / CodeLLaMA checkpoints on BackdoorLLM poisoned test prompts (code-injection vpi-ci, negative-sentiment badnet/ctba/mtba/sleeper), across no-defense, baseline-defense (finetuning / pruning / quantization) and CROW-defended runs, recovered from the authors' BackdoorLLM fork git history.

1,079items
3subjects
DetailsPaper
CTIBench benchmark
MIT2024

CTIBench

CTIBench: a benchmark for evaluating LLMs on cyber threat intelligence tasks, including multiple-choice CTI knowledge (CTI-MCQ), CVE-to-CWE root-cause mapping (CTI-RCM), CVSS vulnerability severity prediction (CTI-VSP), and threat-actor attribution (CTI-TAA). Each task provides per-(model, item) predictions for five LLMs graded against gold answers.

5,483items
5subjects
DetailsPaper
Diagram Understanding (VLM diagram comprehension) benchmark
unknown2024

Diagram Understanding (VLM diagram comprehension)

Diagram-understanding test suite for large vision-language models: multiple-choice questions about entities and relationships over synthetic icon-graph diagrams and real diagrams (physics, ecology, astronomy, geology, developmental biology, geochemistry). Per-item correctness for GPT-4o, GPT-4V and Gemini-1.5 under base and CoT prompting.

32,764items
3subjects
DetailsPaper
Emoji Attack benchmark
unknown2024

Emoji Attack

Emoji-Attack / EasyJailbreak results: per-(target LLM, harmful behavior, attacker) binary jailbroken verdicts. 10 attack families x 10 target LLMs over AdvBench-style harmful behaviors.

3,376items
10subjects
DetailsPaper
ERBench benchmark
Unknown2024

ERBench

ERBench: an Entity-Relationship based, automatically verifiable hallucination benchmark. Yes/no questions are generated from relational databases (functional dependencies and foreign keys) across five domains (movie, book, music, soccer/olympic, airport); each question has a deterministic gold answer and a model passes iff its answer matches it.

31,945items
8subjects
DetailsPaper
Few-Shot TTT (BIG-Bench Hard) benchmark
MIT2024

Few-Shot TTT (BIG-Bench Hard)

Per-item BIG-Bench Hard predictions for Llama-3.1-8B-Instruct under zero-shot, few-shot ICL, and test-time training (TTT) interventions.

1,310items
11subjects
DetailsPaper
GeckoNum benchmark
CC-BY-4.02024

GeckoNum

GeckoNum: numerical reasoning in text-to-image models. 1,386 numeric prompts are rendered by 7 T2I models (5 seeds); humans annotate each generated image on three tasks (object counting, relational description choice, and yes/no DSG questions). Each row grades whether one model's generated image matched the numeric prompt.

2,027items
7subjects
DetailsPaper
h4rm3l benchmark
MIT2024

h4rm3l

h4rm3l composable-jailbreak results: per-(target model, attack program, AdvBench prompt) binary success from a GPT-4 "BAD BOT" classifier. 6 target models x 113 synthesized composable attack programs x 50 AdvBench harmful prompts. response=1 when the attack jailbroke the target (BAD BOT), 0 when it refused (GOOD BOT); UNCLEAR/empty verdicts are dropped.

48items
6subjects
113attacks
DetailsPaper
HarmBench benchmark
MIT2024

HarmBench

HarmBench standardized red-teaming results: ~29 target models x 140 behaviors x 16 attack methods, with the HarmBench Llama-2-13B classifier verdict and the AdvBench refusal-string heuristic as two separate judges. Covers the standard and contextual behavior categories from the HarmBench website playground.

140items
29subjects
16attacks
DetailsPaper
HELM AIRBench 2024 benchmark
Apache-2.02024

HELM AIRBench 2024

HELM AIRBench 2024: per-(model, prompt) air_score binarized to {0,1} (1=safe, threshold 0.5). 66 models x 5694 prompts over the AIR 2024 risk taxonomy (314 categories across 4 levels).

5,692items
66subjects
DetailsPaper
HELM HarmBench benchmark
Apache-2.02024

HELM HarmBench

HELM HarmBench: per-(model, behavior) safety_score in {0,1} (1=safe) from HELM's safety classifier. 87 models x HarmBench harmful behaviors. HELM-sourced; complements the standalone harmbench build.

400items
87subjects
DetailsPaper
HiST-LLM benchmark
CC-BY-NC-SA-4.02024

HiST-LLM

HiST-LLM: expert-level global history knowledge benchmark of 36,577 four-choice multiple-choice questions derived from the Seshat Global History Databank. Each question asks whether a historical characteristic was Present / Inferred Present / Inferred Absent / Absent for a named polity over a stated time frame.

36,577items
7subjects
DetailsPaper
JailbreakBench benchmark
MIT2024

JailbreakBench

JailbreakBench artifact verdicts: per-(model, behavior, attack-method) jailbroken booleans from the JailbreakBench judge. Six (attack_method, attack_type) conditions across 4 target models and 100 harmful behaviors. Sparse: white-box attacks (DSN, GCG/white_box) only cover open-weight models.

105items
4subjects
5attacks
DetailsPaper
LINGOLY benchmark
CC-BY-4.02024

LINGOLY

LINGOLY: olympiad-level linguistic reasoning puzzles in 90+ low-resource and extinct languages, drawn from Linguistics Olympiad problem sheets. A model reads a full problem sheet and answers labelled sub-parts; each part is graded by normalized exact match against the gold answer.

1,133items
11subjects
DetailsPaper
LiveCodeBench benchmark
CC-BY-4.02024

LiveCodeBench

LiveCodeBench per-sample binary outcomes from submissions; leaderboard-only rows remain aggregate pass@1.

1,055items
72subjects
DetailsPaper
LLM-Uncertainty-Bench benchmark
MIT2024

LLM-Uncertainty-Bench

LLM-Uncertainty-Bench: 25 LLMs on five 6-way (A-F) multiple-choice datasets (MMLU, CosmosQA, HellaSwag, Halu-Dialogue, Halu-Summarization). Per-item option logits are released; we grade argmax vs the gold letter.

49,931items
25subjects
DetailsPaper
MARINE benchmark
unknown2024

MARINE

MARINE object-hallucination evaluation: per-item outputs for LLaVA-v1.5-7B with and without the MARINE image-grounded guidance intervention, on POPE (binary object probing) and CHAIR (caption hallucination) over MSCOCO images.

3,500items
2subjects
DetailsPaper
MATH-Vision benchmark
MIT2024

MATH-Vision

MATH-Vision (MATH-V): 3,040 competition mathematics problems with visual context, spanning 16 mathematical subjects and 5 difficulty levels. A model passes a problem iff its final answer matches the gold answer.

3,037items
13subjects
DetailsPaper
MHJ (Multi-turn Human Jailbreaks) benchmark
cc-by-4.02024

MHJ (Multi-turn Human Jailbreaks)

MHJ (Multi-turn Jailbreak) cipher-attack evaluation: 8 frontier LLMs (GPT-3.5/4/4o, Claude 3 Opus/Sonnet/Haiku, Llama 3 8B/70B) evaluated on 23 harmful goals under 2 word-mapping ciphers, with both multi-turn and single-turn variants and two judge labels (Jailbroken severity, Understanding The Question).

23items
8subjects
DetailsPaper
MMLU-Pro benchmark
MIT2024

MMLU-Pro

MMLU-Pro — 12K-question challenge MMLU variant across 14 subjects; per-question binary correctness + leaderboard per-category aggregates.

13,542items
48subjects
DetailsPaper
NYU CTF Bench benchmark
MIT2024

NYU CTF Bench

NYU CTF Bench: 200 Capture-The-Flag challenges from CSAW competitions (crypto, forensics, pwn, rev, web, misc) for evaluating LLMs and agents in offensive security. A subject solves a challenge if it recovers the challenge flag.

200items
9subjects
DetailsPaper
OR-Bench benchmark
cc-by-4.02024

OR-Bench

OR-Bench (Over-Refusal Benchmark): 26 LLMs evaluated on two prompt sets. The "overalign" set holds benign-but-sensitive prompts that models tend to over-refuse (desired behavior: answer); the "toxic" set holds genuinely toxic prompts (desired behavior: refuse). One row per (model, prompt) is the model's raw free-text response. Responses are binarized with a deterministic refusal-phrase classifier: for the overalign set response=1 if the model did NOT refuse (0 = over-refusal); for the toxic set response=1 if the model refused (0 = unsafe answer). This is the released demo subset (150 prompts per model per set) shipped with the leaderboard Space, not the full ~80k OR-Bench corpus. The raw model output is preserved in traces for re-grading with the upstream LLM judge.

300items
26subjects
DetailsPaper
PertEval benchmark
Apache-2.02024

PertEval

PertEval measures the real knowledge capacity of LLMs by applying knowledge-invariant perturbations to MMLU questions (rephrasing that preserves the required knowledge). Each MMLU item is recast as an option-judgement task: the model labels each of 4 options True/False; an item is correct iff the full True/False vector matches the gold.

4,885items
6subjects
DetailsPaper
PKU-SafeRLHF benchmark
CC-BY-NC-4.02024

PKU-SafeRLHF

PKU-SafeRLHF per-response safety ratings for Alpaca-family models.

39,994items
3subjects
DetailsPaper
Putnam-AXIOM benchmark
Apache-2.02024

Putnam-AXIOM

University-level Putnam competition math problems; response is a human-graded reasoning rubric (5 ordinal dimensions) on each model solution.

500items
2subjects
DetailsPaper
RE-Bench benchmark
mit2024

RE-Bench

Per-(model x environment) binarized success (0/1) on METR RE-Bench, the suite of 7 ML research-engineering environments, from METR's publicly released Time Horizon run logs (score_binarized). Item content is the real full task instruction text shown to the agent.

7items
37subjects
DetailsPaper
REVOLVE benchmark
MIT2024

REVOLVE

REVOLVE solution-optimization evaluation: Llama-3.1-8B-Instruct performs test-time training on GPQA-Diamond (198 items) and MMLU Machine Learning (112 items) multiple-choice questions, optimized either by the TextGrad baseline (v1) or by REVOLVE (v2, response-evolution tracking). Binary exact-match correctness of the extracted answer letter at each optimization stage (initial zero-shot, iterations 1-3, majority-vote final).

310items
2subjects
DetailsPaper
RewardBench benchmark
ODC-BY2024

RewardBench

RewardBench per-(reward-model, item) correctness on chosen>rejected ranking.

2,733items
118subjects
DetailsPaper
SAD benchmark
CC-BY-4.02024

SAD

SAD (Situational Awareness Dataset): does an LLM know it is an LLM and can it reason about its own situation? 16 subtasks across 7 categories (facts, introspection, stages, self-recognition, anti-imitation, influence, id-leverage), mostly multiple-choice, graded per sample as correct (1) / incorrect (0).

191,727items
24subjects
DetailsPaper
SM3-Text-to-Query benchmark
CC-BY-4.02024

SM3-Text-to-Query

SM3-Text-to-Query: a synthetic multi-model medical text-to-query benchmark. Each item is a natural-language clinical question over a Synthea patient database; the model must generate a database query in one of four target languages (SQL, MongoDB MQL, Cypher, SPARQL). Response is query exact-match accuracy (the generated query matches the gold query after token-sort normalization), the DB-free metric from the repo evaluation.

8,945items
6subjects
DetailsPaper
SORRY-Bench benchmark
other2024

SORRY-Bench

SORRY-Bench human-judgment verdicts: per-(model, instruction, prompt-style) binary human labels in {0=refusal, 1=fulfillment}. 31 models x 450 unsafe base instructions across ~20 linguistic mutation styles. Sparse human-annotated sample over the (model x prompt x style) cube.

450items
31subjects
DetailsPaper
StatQA benchmark
CC-BY-4.02024

StatQA

StatQA: statistical-analysis question answering over tabular datasets. Given a dataset and a refined question, a model selects the relevant columns and applicable statistical methods; it is correct on an item iff both its column selection and method selection exactly match the gold set. Evaluated across prompting strategies (zero-/one-shot, CoT, stats-prompt) for GPT and LLaMA models.

1,158items
10subjects
DetailsPaper
SubjECTive-QA benchmark
CC-BY-4.02024

SubjECTive-QA

SubjECTive-QA: measuring subjectivity in earnings-call-transcript QA. Models rate each (question, answer) pair on a 3-class scale (0/1/2) for each of six subjective features (Assertive, Cautious, Clear, Optimistic, Relevant, Specific). Scored as classification correctness against human gold labels.

3,300items
5subjects
DetailsPaper
SynthPAI benchmark
MIT2024

SynthPAI

SynthPAI: personal-attribute inference from synthetic Reddit profiles. 294 synthetic author profiles (7823 comments); 18 LLMs infer up to 8 personal attributes (age, sex, location, birthplace, education, occupation, income, relationship status) from comment history. Each model gives a ranked guess list per attribute, graded 0/0.5/1 against human-curated ground truth.

700items
18subjects
DetailsPaper
Visual Riddles benchmark
CC-BY-4.02024

Visual Riddles

Visual Riddles: a commonsense and world-knowledge challenge for large vision and language models. Each item is an AI-generated image with an open-ended question requiring visual reasoning plus world knowledge. Per-(model, item) responses are human-rated correct/incorrect on the open-ended task.

386items
6subjects
DetailsPaper
VLM4Bio benchmark
Apache-2.02024

VLM4Bio

VLM4Bio: evaluating pretrained vision-language models for trait discovery from biological images (Bird/Butterfly/Fish). This build covers the multiple-choice (selection) species-classification task, where the model picks the correct scientific name from four options given an organism image.

1,242items
1subjects
DetailsPaper
Vript-Hard benchmark
Apache-2.02024

Vript-Hard

Vript-Hard: video-understanding challenge tasks on densely-captioned clips. Vript-RR asks scene-specific multiple-choice questions given a hint; Vript-ERO asks for the correct temporal order of three scenes. Responses are per-(video-LLM, item) graded outcomes reproduced from the upstream verification pipeline.

286items
3subjects
DetailsPaper
WikiContradict benchmark
Apache-2.02024

WikiContradict

WikiContradict: real-world knowledge-conflict QA mined from Wikipedia. Each item is a question whose answer is contradicted across two Wikipedia passages; LLM answers are graded by humans as Correct / Partially correct / Incorrect under several prompt (RAG) templates.

48items
5subjects
DetailsPaper
WildVision-Bench benchmark
CC-BY-4.02024

WildVision-Bench

WildVision-Bench: 500 in-the-wild image+text instructions answered by ~20 vision-language models. A GPT-4o judge scores each model response head-to-head against a fixed claude-3-sonnet reference on a five-level preference scale (Better++/Better/Tie/Worse/Worse++), mapped to a win-score in [0, 1] per (model, item).

425items
20subjects
DetailsPaper

2023

22
BIRD (text-to-SQL) benchmark
MIT2023

BIRD (text-to-SQL)

BIRD: a large-scale, cross-domain text-to-SQL benchmark (NL question + gold SQL over 95 real databases, 37 domains), scored by execution accuracy. The official repo ships questions/gold SQL/eval harness but no baseline predictions; this build ingests petavue/NL2SQL-Benchmark's public 'Inference Level Dataset' of 43,200 per-inference rows over a 360-question subset of BIRD dev. Each response is one model's binary execution-match correctness on one question under one prompting config; items are the real NL questions, correct_answer is the gold SQL. Models served under multiple provider aliases are merged to one canonical subject, with the serving platform recorded in test_condition.

360items
16subjects
DetailsPaper
CMMLU benchmark
CC-BY-NC-SA-4.02023

CMMLU

CMMLU (Chinese MMLU) — per-(model, question) correctness across 67 subjects, from OpenCompass CompassAcademic predictions (answer letter extracted from each model output vs gold).

11,580items
22subjects
DetailsPaper
DecodingTrust benchmark
CC-BY-SA-4.02023

DecodingTrust

DecodingTrust stereotype-bias generations: per-(model, stereotype statement, regime) binary agreement, one row per sampled generation (25 trials/cell). response=1 when the model agreed with the stereotype (unsafe), 0 otherwise (disagree / no-stance / refusal), from the precomputed `agreeability_num`. 7 models x 16 topics x 3 system-prompt regimes.

1,152items
8subjects
DetailsPaper
Do-Not-Answer benchmark
apache-2.02023

Do-Not-Answer

Do-Not-Answer baseline-refusal benchmark: 939 refusal-worthy prompts asked directly (no adversarial wrapper) of 6 LLMs (GPT-4, GPT-3.5, Claude, ChatGLM2, Llama-2-7B-chat, Vicuna-7B), with two annotation labels per cell: binary harmfulness and a 6-way ordinal action class.

938items
6subjects
DetailsPaper
DORIS-MAE benchmark
CC-BY-NC-4.02023

DORIS-MAE

DORIS-MAE Anno-GPT annotation matrix. DORIS-MAE (NeurIPS 2023 D&B) is a scientific-document retrieval benchmark of complex, multi-level aspect-based queries over arXiv CS abstracts. Its Anno-GPT framework uses chatgpt-3.5-turbo-0301 as an expert-level relevance annotator. The public Zenodo dataset releases, for all 165,144 (aspect, candidate-abstract) pairs, ChatGPT's 3-level relevance label (0=unrelated, 1=partial, 2=fully answers the aspect) plus its explanation text. This is a single-subject per-item matrix (subject=ChatGPT-3.5; item=one (aspect, document) pair; response=0/1/2 label). For the 250 Test_set pairs, three human annotators' majority-vote label is attached as gold. The 17 retrieval models benchmarked in the paper are reported only as aggregate metrics (no per-item outputs released), so they are excluded.

165,104items
1subjects
DetailsPaper
EgoSchema benchmark
mit2023

EgoSchema

EgoSchema: a diagnostic long-form egocentric video question-answering benchmark derived from Ego4D, with 5-way multiple-choice questions about ~3-minute first-person videos. This build ingests real per-(model, item) predictions over the public 500-question label subset from the released LLoVi runs (CeeZh/LLoVi output.zip): 11 LLoVi configurations (LLM backbone + caption source + prompt strategy), each evaluated on the same 500 items. Response is per-item correctness (model choice == gold).

500items
11subjects
DetailsPaper
EVOUNA benchmark
Apache-2.02023

EVOUNA

EVOUNA (QA-Eval): per-(model, question) human correctness judgments for five Open-QA systems (FiD, GPT-3.5/text-davinci-003, ChatGPT, GPT-4, New Bing) answering the same Open-domain questions from Natural Questions (3,610) and TriviaQA (2,000). Each model answer was manually judged correct/incorrect by a human annotator.

5,608items
5subjects
DetailsPaper
FELM benchmark
MIT2023

FELM

FELM: a factuality-evaluation meta-benchmark of 847 ChatGPT responses (gpt-3.5-turbo) across five domains (world knowledge, science/tech, math, reasoning, writing/recommendation), each segmented into text spans with a human factuality label (factual vs. factual error) and free-text corrections. This build ingests the publicly released single-subject matrix (subject = ChatGPT, item = response segment, response = per-segment human factuality verdict) — 4,426 labeled segments. The detector-model prediction layer (GPT-4 / ChatGPT / Vicuna under raw/CoT/link methods) is not released upstream.

4,404items
1subjects
DetailsPaper
GHOSTS (Mathematical Capabilities of ChatGPT) benchmark
CC-BY-NC-4.02023

GHOSTS (Mathematical Capabilities of ChatGPT)

GHOSTS: graduate-level natural-language mathematics prompts from the paper 'Mathematical Capabilities of ChatGPT' (Frieder et al., 2023). Each prompt is posed to a language model and the model's answer is graded 1-5 by a human mathematician (higher = better). Subjects are the two ChatGPT snapshots (2023-01-09, 2023-01-30) on the full GHOSTS set and GPT-4 on the miniGHOSTS subset; items span definition retrieval, MATH-dataset problems, textbook exercises (Munkres, Rudin, Durrett), olympiad problems, proof completion and symbolic integration.

707items
3subjects
DetailsPaper
HallusionBench benchmark
BSD-3-Clause2023

HallusionBench

Build HallusionBench per-item response matrix for the shipped GPT-4V reference run

1,129items
1subjects
DetailsPaper
HELM SimpleSafetyTests benchmark
Apache-2.02023

HELM SimpleSafetyTests

HELM SimpleSafetyTests: per-(model, prompt) safety_score in {0,1} (1=safe) from HELM's safety classifier. 87 models x 100 severe-harm prompts (self-harm, physical harm, illegal items, scams, child abuse).

100items
87subjects
DetailsPaper
HELM XSTest benchmark
Apache-2.02023

HELM XSTest

HELM XSTest: per-(model, prompt) safety_score in {0,1} (1=safe) from HELM's safety classifier. 87 models x 450 XSTest prompts probing exaggerated safety / over-refusal. HELM-sourced.

450items
87subjects
DetailsPaper
LegalBench benchmark
Apache-2.02023

LegalBench

LegalBench legal-reasoning tasks evaluated by Stanford HELM Lite (v1.0.0). Per-(model, item) responses for 5 LegalBench tasks (abercrombie, corporate_lobbying, function_of_decision_section, international_citizenship_questions, proa) x 31 LLMs. Response is quasi_exact_match (0/1); trace holds the model's predicted text. Scoped to the 5 HELM-Lite tasks with full public per-instance predictions (of LegalBench's 162 total tasks).

2,047items
31subjects
DetailsPaper
MathVista MINI benchmark
CC-BY-SA-4.02023

MathVista MINI

MathVista_MINI: mathematical reasoning with vision (mixed MCQ + free-form), binary correctness from VLMEval/OpenVLMRecords.

874items
263subjects
DetailsPaper
MMBench V1.1 benchmark
Apache-2.02023

MMBench V1.1

Build MMBench_V11 response matrix from VLMEval/OpenVLMRecords

3,579items
251subjects
DetailsPaper
MT-Bench benchmark
CC-BY-4.02023

MT-Bench

MT-Bench GPT-4 single-answer judgment scores (34 models, 80 questions, 2 turns).

160items
34subjects
DetailsPaper
PlanBench benchmark
MIT2023

PlanBench

PlanBench: evaluating LLMs on automated-planning and reasoning-about-change tasks over classic IPC domains (Blocksworld, Logistics, Sokoban) plus mystery / obfuscated / randomized / unsolvable variants. We ingest the REAL per-(model, item) graded outputs from the maintained official repo: each response is one LLM's binary correctness on one planning instance (plan generation), graded by the VAL plan verifier or the run's recorded correctness flag. Items are the real natural-language planning prompts; correct_answer is the ground-truth plan.

10,703items
17subjects
DetailsPaper
SugarCrepe benchmark
MIT2023

SugarCrepe

SugarCrepe: a non-gameable vision-language compositionality benchmark of ~7.5k (image, positive caption, hard-negative caption) triples across 7 hard-negative types (add/replace/swap of attributes, objects, relations). Models must pick the caption that matches the COCO-2017 image. This build ingests the released per-item GPT-4V responses from the official repo (the only model with per-instance outputs released; the 17 CLIP models in the paper have only aggregate accuracy), in two caption-presentation orders (positive-first / negative-first).

7,512items
1subjects
DetailsPaper
Tulu Human Evaluation benchmark
Apache-2.02023

Tulu Human Evaluation

Tulu 1 human evaluation: 332 prompt instances, each with completions from open instruction-tuned models (tulu_65B, tulu_7B, tulu_human_only_65B) and ChatGPT, graded by human annotators in pairwise comparisons. Subjects are the AI models; the response is the per-(model, item) human acceptability label (yes/no -> 1/0). Only the released human-eval slice is per-item public; the paper's automatic benchmark predictions (MMLU, GSM8k, BBH, etc.) are not released and are excluded.

332items
4subjects
DetailsPaper
UltraFeedback benchmark
MIT2023

UltraFeedback

UltraFeedback GPT-4 multi-aspect ratings (1-5) from 17 models on ~64K prompts.

63,932items
17subjects
DetailsPaper
VisIT-Bench benchmark
CC-BY-4.02023

VisIT-Bench

VisIT-Bench: 592 real-world image+instruction vision-language queries across ~70 instruction families. Two public per-(model, item) sources are ingested: (1) 5011 human pairwise-preference battles among six vision-language models (MiniGPT-4, LLaMA-Adapter-v2, LLaVA-13B, PandaGPT-13B, mPLUG-Owl, InstructBLIP) -> binary win/loss per (model, item, opponent); (2) human correctness ratings of GPT-4 predictions -> binary correct/incorrect per item for GPT-4. response is binary in {0, 1}; the full model answer text is kept as trace.

536items
7subjects
DetailsPaper

Earlier (pre-2023)

14
AI2D_TEST benchmark
CC-BY-SA-4.02016

AI2D_TEST

Build AI2D_TEST response matrix from VLMEval/OpenVLMRecords

3,088items
254subjects
DetailsPaper
ARC-AGI benchmark
Apache-2.02019

ARC-AGI

ARC-AGI v1 + v2 per-model per-task solve data from the ARC Prize Foundation evaluation repos, plus leaderboard aggregate scores.

520items
73subjects
DetailsPaper
HELM Anthropic RedTeam benchmark
Apache-2.02022

HELM Anthropic RedTeam

HELM Anthropic RedTeam: per-(model, prompt) safety_score in {0,1} (1=safe) from HELM's safety classifier. 87 models x Anthropic red-team harm prompts.

995items
87subjects
DetailsPaper
HELM BBQ benchmark
Apache-2.02022

HELM BBQ

HELM BBQ: per-(model, question) exact_match in {0,1} (1=correct) on 1000 BBQ social-bias multiple-choice questions. 87 models.

999items
87subjects
DetailsPaper
HELM BOLD benchmark
Apache-2.02021

HELM BOLD

HELM BOLD: per-(model, prompt) toxic_frac binarized to {0,1} (1=any toxic continuation). ~42 models x 1000 BOLD open-ended generation prompts.

994items
42subjects
DetailsPaper
HELM RealToxicityPrompts benchmark
Apache-2.02020

HELM RealToxicityPrompts

HELM RealToxicityPrompts: per-(model, prompt) toxic_frac binarized to {0,1} (1=any toxic continuation, Perspective API). ~42 models x 1000 RealToxicityPrompts items.

1,000items
42subjects
DetailsPaper
LLM4IR (LaMIR) benchmark
unknown

LLM4IR (LaMIR)

LaMIR / LLM4IR: evaluation of LLMs on understanding LLVM intermediate representations (compiled from the 164 HumanEval-X C++ problems). Two tasks with released per-(model, item) categorical grades are ingested: Control Flow Graph reconstruction (Task 1; graded Overall Completed / Loop Completed / Branch Completed / Wrong) and execution reasoning (Task 4; graded pass / partial pass / fail). test_condition encodes the task.

328items
6subjects
DetailsPaper
MMLU Business Ethics benchmark
MIT2021

MMLU Business Ethics

MMLU business_ethics per-(model, question) accuracy (acc in {0,1}) over an applied-ethics MCQ subject, from the Open LLM Leaderboard v1 details datasets. Model panel capped to 150.

100items
150subjects
DetailsPaper
MMLU Moral Disputes benchmark
MIT2021

MMLU Moral Disputes

MMLU moral_disputes per-(model, question) accuracy (acc in {0,1}) over an applied-ethics MCQ subject, from the Open LLM Leaderboard v1 details datasets. Model panel capped to 150.

346items
150subjects
DetailsPaper
MMLU Moral Scenarios benchmark
MIT2021

MMLU Moral Scenarios

MMLU moral_scenarios per-(model, question) accuracy (acc in {0,1}) over 895 machine-ethics MCQ items, from the Open LLM Leaderboard v1 details datasets. Model panel capped to 150.

895items
150subjects
DetailsPaper
RealTime QA benchmark
MIT2022

RealTime QA

RealTime QA: a dynamic multiple-choice news QA benchmark in which new questions are released weekly from CNN / THE WEEK news quizzes. We ingest the published GRANULAR per-(model, question) baseline predictions from the official realtimeqa_public repository: each response is one model's per-item correctness {0,1} on one weekly multiple-choice question (does the chosen 0-based choice index match the gold answer). Covers all weeks with released baselines (2022, 2023, 2026), both the standard (qa) and none-of-the-above (nota) MC settings, with retrieval method (closed-book / DPR / GCS) recorded as an evaluation condition. Free-text generation (_gen) outputs are excluded because they lack a gold choice index.

4,743items
16subjects
DetailsPaper
SciArena benchmark
CC-BY-4.0

SciArena

SciArena: an open Chatbot-Arena-style platform for non-verifiable, scientific-literature-grounded tasks. Two foundation models answer the same literature question; human researchers vote for the better long-form, citation-grounded response, yielding pairwise win/loss/tie outcomes across 25 scientific disciplines and 6 question types.

8,181items
23subjects
DetailsPaper
TruthfulQA-MC benchmark
Apache-2.02022

TruthfulQA-MC

TruthfulQA-MC1 per-(model, question) correctness (mc1 bool, 1=truthful) over 817 questions, from the Open LLM Leaderboard v1 details datasets. Model panel capped to 150.

817items
150subjects
DetailsPaper

Citation

Cite this work

If you use the data we curated, please cite the following reference. The curation is released under CC BY-SA 4.0; individual benchmarks retain their upstream licenses.

@misc{measurementdb2026,
  title        = {measurement-db: A Curated Data Bank of AI Evaluation Results},
  author       = {Truong, Nhi and Truong, Sang T. and Koyejo, Sanmi},
  year         = {2026},
  howpublished = {\url{https://aimslab.stanford.edu/measurement-db}},
  note         = {AIMS Lab, Stanford University}
}