Skip to main content
AIMS — AI Measurement Science

Search AIMS

Find pages, publications, events, course projects, and software. Results update as you type; press Enter to open the first result.

Discover, share, and analyze fine-grained AI measurement data

The Measurement Data Bank is a curated exchange of item-level AI evaluation data: what each system was tested on, how it responded, and how each response was scored. Analyze those measurements with torch_measure.

View schemaAnalyze with torch_measure
Contribute dataDownload dataCite this work
Benchmarks
270
Items
8.4M
AI subjects
781
Responses
92.1M

Benchmark catalog

Benchmark type

Responses are scored by deterministic or programmatic verifiers and test harnesses.

Automated selected. 264 benchmarks available.

264 of 264 automated benchmarks
Meta-analyses

2026

35
PU

AGC-Bench

Released model attempts and individual creativity judgments across the AGC-Bench component datasets.

items
8,600items
AI subjects
83AI subjects
U

Agentic Review Systems — Perturbation Benchmark

Released early per-error detection results for a Claude Opus 4.6 review workflow on six perturbed papers. Items contain the released paper text; grading criteria identify the injected errors. This is a partial experimental slice, not the full paper evaluation.

items
105items
AI subjects
3AI subjects
R

Auditing Sabotage Bench

Recorded AI audits of honest and sabotaged machine-learning research codebases and their papers.

items
68items
AI subjects
18AI subjects
R

AutoResearchBench

Released agent attempts at identifying individual scientific papers and collecting sets of relevant papers.

items
60items
AI subject
1AI subject
TL

BenGER

Recorded language-model attempts at German legal decision and case-solution tasks, with explicit grading protocols.

items
760items
AI subjects
41AI subjects
P

CEO-Bench

Language-model agents manage a simulated startup; retain published outcomes and complete released trajectories.

items
3items
AI subjects
18AI subjects
K

Classroom AI

Readability of educational answers from base, grade-prompted and grade-finetuned GPT models. Preserve the released DGPT answers and integrate their recorded readability estimates with the authors' classifier.

items
734items
AI subjects
14AI subjects
SK

CoffeeBench

Recorded net income of language-model agents managing a coffee roaster in a 90-day multi-agent economy.

item
1item
AI subjects
7AI subjects
S

DataClawBench

Released accuracy judgments for eight OpenClaw agents on 492 exploratory financial data-analysis tasks.

items
492items
AI subjects
8AI subjects
G

EDU-CIRCUIT-HW

Released model transcriptions of 513 anonymized circuit-analysis homework samples and the available recognition-error judgments for six MLLMs.

items
513items
AI subjects
6AI subjects
UM

ExploitGym

Published per-task on-target verdicts for five ExploitGym agent submissions, retaining mitigation profiles and the additional two-hour result view.

items
1,737items
AI subjects
5AI subjects
SM

FrontierOR

Evaluates whether models can translate operations-research problems into executable optimization programs that produce feasible solutions.

items
180items
AI subjects
15AI subjects
S

HealthAdminBench

Computer-use agent results on 135 synthetic healthcare-administration tasks, with original strict and process-check-removed judgments.

items
540items
AI subjects
7AI subjects
P

IKP

Evaluates factual recall across seven tiers of obscurity, using an LLM judge to distinguish correct answers, incorrect answers, and refusals.

items
1,400items
AI subjects
170AI subjects
I

Legal RAG Bench

End-to-end legal RAG benchmark: 100 Victorian criminal-law questions x (3 embedders x 2 frontier LLMs), judged on binary correctness, groundedness, and retrieval accuracy.

items
300items
AI subjects
6AI subjects
I

Live-Agent Risk

Published multiplayer Risk games with per-seat win/loss outcomes, recorded player configurations and full available interaction logs.

items
73items
AI subjects
31AI subjects
E

MathArena Platform

Published model attempts on ArXivMath, ArXivLean, BrokenArXiv and USAMO 2026, with original requests, full conversations and native grading scales.

items
1,414items
AI subjects
68AI subjects
G

MathConstraint

Adaptive combinatorial-reasoning benchmark: constraint-satisfaction (CSP/SAT) decision problems with solver-verified SAT/UNSAT ground truth; 12 models evaluated with and without a sandboxed SAT/SMT solver.

items
504items
AI subjects
12AI subjects
C

MyPCBench

Recorded rubric-level LLM judgments and original viewer traces from the 15 official MyPCBench showcase trajectories.

items
118items
AI subjects
6AI subjects
T

NatureBench

Published Match-SOTA outcomes for 23 coding-agent configurations on 90 scientific ML tasks, with complete available native run files and recorded validity judgments.

items
181items
AI subjects
23AI subjects
U

OpenBioRQ

OpenBioRQ: 657-question core of unsolved biomedical research questions for tool-using agents, graded by a frozen per-question checklist; solve = checklist score >= 0.5. Full per-(model x item) matrix released for 11 agents (roster, held-out, frontier, no-tool ablations).

items
657items
AI subjects
11AI subjects
HN

PerfCodeBench

Published correctness-and-runnability assessments of generated code replacements for systems optimization tasks, with complete source inputs and native execution logs.

items
1,854items
AI subjects
20AI subjects
MS

ProgramBench

Published ProgramBench attempts to reconstruct executable behavior from a compiled program and its documentation, with native fractional behavioral-test scores and complete recorded agent trajectories.

items
1,317items
AI subjects
28AI subjects
UH

Real-POCQi

Real-POCQi: blinded specialty-matched physician pairwise evaluation of clinical AI systems (GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.8, OpenEvidence) on 620 real point-of-care queries, rated on five dimensions.

items
3,100items
AI subjects
4AI subjects
UK

RelianceScope

Recorded LLM classification of help-seeking and response-use in student–chatbot interaction segments.

items
1,200items
AI subjects
24AI subjects
S

ResearchClawBench

Recorded autonomous scientific research attempts, graded against expert-weighted task rubrics.

items
40items
AI subjects
37AI subjects
U

Robust Reasoning Benchmark

Recorded mathematical reasoning and text-reconstruction attempts under deterministic prompt transformations.

items
4,895items
AI subjects
31AI subjects
U

S-GRADES

Released model predictions for student-response assessment across the S-GRADES dataset families.

items
40,509items
AI subjects
77AI subjects
B

SketchJudge

SketchJudge (arXiv 2601.06944): 16 multimodal LLMs act as graders of 1,015 hand-drawn student STEM answers (geometry, physics, chart, flowchart). response = binary grading accuracy, 1 iff the model's correctness verdict matches the expert gold label, under WithRef/NoRef settings and baseline/CoT/rubric prompt styles.

items
4,053items
AI subjects
38AI subjects
J

Software World Models (DL4C'26)

Coding LLMs predict a runtime test outcome (passed/failed/error) for 435 library-level cases derived from SWE-bench Verified; response is per-item outcome-prediction correctness.

items
434items
AI subjects
18AI subjects
MT

WeaveBench

Selected hybrid computer-use trajectories from the original WeaveBench gallery, with complete published task prompts, grading evidence and linked screenshots.

items
114items
AI subjects
7AI subjects
I

WorkBench Revisited

Evaluates tool-using agents on workplace tasks by comparing their final sandbox state with the expected outcome.

items
690items
AI subjects
24AI subjects

2025

80
X

Abstract-Reason

Abstract-Reason benchmark: rule-based symbolic reasoning tasks (basic/extended computation, non-decimal number bases, symbolic function inference, symbolic manipulation) with systematic symbol remapping to separate genuine abstraction from memorization.

items
9,089items
AI subject
1AI subject
NU

AdaptiveStep candidate pools

Released math and code candidate solutions used for AdaptiveStep Best-of-N evaluation. The tables measure individual policy-generator attempts, not reward-model selection or token-guided decoding.

items
4,525items
AI subjects
3AI subjects

AEGIS 2.0 response safety

Assesses the safety of recorded Mistral responses using released annotations.

items
11,500items
AI subject
1AI subject

AGI-Elo

Item-level MMLU predictions released for AGI-Elo, a study of joint task difficulty and AI-system competency. This curation covers the released question-answering observations, not derived Elo ratings.

items
13,953items
AI subjects
20AI subjects
UM

AlgoTune

Evaluates coding agents on producing correct numerical programs that match or improve reference runtime.

items
154items
AI subjects
18AI subjects
A

Alignment Faking (RL)

Published judgments of alignment-faking reasoning across model organisms, interventions and reinforcement-learning checkpoints in Anthropic's training-time mitigation study.

items
1,464items
AI subjects
4,278AI subjects
H

Alpha-SQL (BIRD dev)

Released final SQL queries from Alpha-SQL, a Monte Carlo tree-search system using Qwen2.5-Coder-32B-Instruct, on the BIRD development questions.

items
1,534items
AI subject
1AI subject

AVerImaTeC

Released AIC CTU development predictions for image-text claim verification, measured by the upstream verdict-correctness component.

items
152items
AI subject
1AI subject

BRACE (CAF-Score decisions)

Released CAF-Score caption-pair decisions on BRACE audio: correctness of smoothed and raw score variants against the annotations actually used in the third-party evaluation.

items
12,631items
AI subjects
28AI subjects

CellVerse

Cell-type annotation from verbalized single-cell gene expression. This curation covers the released DeepSeek-R1 results for the 748 multiple-sclerosis scRNA-seq questions.

items
748items
AI subject
1AI subject

ChartMuseum

Chart question answering with the 162 released Claude 3.7 Sonnet development outputs and their original images.

items
162items
AI subject
1AI subject

ChiPBench

Published circuit-level placement and synthesis measurements, full released design kits and native metric scales.

items
230items
AI subjects
11AI subjects
P

ConfAgents

ConfAgents medical multiple-choice diagnosis: per-(agent, question) binary correctness (1 = final answer matches the gold option) on a released 16-question subset (8 MedBullets + 8 MedQA) answered by four agent frameworks (ConfAgents, ColaCare, MDAgents, MedAgent).

items
16items
AI subjects
4AI subjects
S

Critic-Discernment Game

Released Lighteval observations for the Llama-3.1-8B-Instruct baseline and a CDG-trained checkpoint on GSM8K and MATH-500.

items
1,819items
AI subjects
4AI subjects
MP

CSEDB

Released clinical answers and LLM assessments from the main, prompt-optimization and repeated-generation experiments.

items
2,069items
AI subjects
7AI subjects

C-SEO Bench

C-SEO Bench measures changes in document citation ranking in conversational search. Each item is a paired Original/intervention prompt with one target document; each response is that document's rank improvement for one released model. The release spans books, retail, videogames, web, news and debate.

items
242,265items
AI subjects
4AI subjects
T

DAS Medical Hallucination

Recorded responses to medical red-teaming prompts and the authors' multi-agent hallucination-detector categories.

items
108items
AI subjects
15AI subjects

DAVE

Measures audio-visual reasoning through questions linking sounds to events in egocentric videos.

items
3,922items
AI subjects
8AI subjects
CU

DIS-CO (MovieTection)

Individual movie-title guesses from four vision-language models, queried with movie frames or their released captions.

items
28,000items
AI subjects
4AI subjects
P

EHRFlowBench

Published completion flags and full answer text for seven agent frameworks on the 20-task HealthFlow review subset, with the original EHR inputs and reference reports.

items
20items
AI subjects
7AI subjects
I

The Elicitation Game

Released WMDP multiple-choice outcomes under prefilling, adversarial-suffix and anti-refusal interventions, with original questions and available answer records.

items
2,960items
AI subjects
14AI subjects

EngiBench

Released EngiOpt Beams2D demonstration results: conditional GAN and conditional diffusion outputs at three explicit design conditions, with initial, cumulative and final signed optimality gaps.

items
9items
AI subjects
2AI subjects

FGBench

Molecular property reasoning at the functional-group level. This curation links individual model outputs to the 7,146-question test bank, preserving classification correctness and numeric squared errors.

items
7,146items
AI subjects
8AI subjects
UC

FineGRAIN (T2I)

Released text-to-image generations on 760 prompts covering 27 failure modes, with human annotations for the five original generators.

items
760items
AI subjects
17AI subjects
U

GenAI Learning

Original student practice and exam grades from a mathematics tutoring study, with separately annotated standalone GPT-4 answers.

items
162items
AI subjects
944AI subjects
EE

HIVMedQA

Published HIV question-answering runs and five clinical judge ratings, retaining reference-guided, reference-free and reference-paraphrasing conditions separately.

items
2,570items
AI subjects
13AI subjects
U

ICPC-2 Code Selector

Evaluates whether models can select a primary-care code for a Brazilian-Portuguese clinical expression from retrieved candidates, or correctly abstain.

items
2,175items
AI subjects
39AI subjects
S

KernelBench

Released kernel-generation correctness assessments for the original 250 PyTorch workloads, including baseline, repeated sampling and iterative refinement.

items
250items
AI subjects
19AI subjects

KRIS-Bench

Released image-editing attempts and their individual consistency, instruction-following, quality and knowledge-plausibility ratings.

items
8,941items
AI subjects
20AI subjects
U

LiveAoPSBench

Released 2024 LiveAoPSBench Olympiad mathematics questions and complete source-reported model correctness matrix.

items
5,323items
AI subjects
13AI subjects
K

LLM4IR (LaMIR)

Published program-level assessments of LLVM control-flow reconstruction, decompilation, summarization and execution reasoning.

items
1,804items
AI subjects
19AI subjects
G

LLM-BP TAG node-classification results

Released GPT-4o text-only node-classification attempts on eleven existing text-attributed graph datasets in the LLM-BP study.

items
152,139items
AI subject
1AI subject
C

LLM survey-simulation responses (EEDI and OpinionQA)

Published synthetic survey responses from eight LLMs on 412 EEDI questions and 385 OpinionQA questions. These are simulated human responses, not direct estimates of the LLMs' question-solving ability.

items
797items
AI subjects
8AI subjects

LLMail-Inject

Released attempts against defended LLM email agents in both LLMail-Inject challenge phases. Subjects are target agents, configured by base model, defense and phase. Items combine submitted emails with the released scenario context; the outcome is attack success, which is undesirable for the target agent.

items
207,291items
AI subjects
23AI subjects
E

MathArena

Measures mathematical problem solving through final-answer correctness and criterion-level proof grades.

items
1,755items
AI subjects
104AI subjects

MedAgentBoard

Released medical QA, lay-summary, EHR prediction and clinical workflow records with exact available assessments and complete native logs.

items
2,301items
AI subjects
83AI subjects
S

MedHELM MedHallu

MedHELM medhallu: per-(model, item) exact_match in {0,1} (1=correct) on 1000 medical hallucination detection: label an answer as hallucinated or faithful. 13 models from the MedHELM v4.0.0 release.

items
2,000items
AI subjects
13AI subjects
U

MedWatermark-FWS

Complete released medical-watermark generations for three models, with cached paired GPT-Judger quality ratings for Meditron and original auxiliary detection statistics.

items
2,376items
AI subjects
72AI subjects

MergeBench

Released safety assessment records for two RegMeanPlusPlus merged language models, with explicit attribution to WildGuardTest, HarmBench, XSTest and Do Anything Now.

items
1,813items
AI subjects
2AI subjects

MLIP Arena

Released per-system observations of machine-learned interatomic potentials across physics-aware simulation tasks.

items
13,706items
AI subjects
71AI subjects
H

MMDocRAG

MMDocRAG tests whether models can answer questions about documents by weaving retrieved text and images into a cited answer, which a judge model rates for fluency, citation quality, coherence, reasoning and factuality.

items
2,000items
AI subjects
68AI subjects
UH

MMOral-Bench recorded policies

Original MMOral closed/open dental radiograph tasks and released OPTG voter/ensemble choice records.

items
1,069items
AI subjects
85AI subjects
U

MorphKV results on LongGenBench

Released LongGenBench constraint judgments and generated outputs for KV-cache interventions studied in MorphKV; these are measurements on LongGenBench, not a new task bank.

items
537items
AI subjects
11AI subjects
M

MORQA evaluator ratings

Released LLM evaluator ratings of medical candidate answers in English and Chinese, with original questions, reference answers and expert annotations.

items
5,443items
AI subjects
4AI subjects
EE

MTBBench

MTBBench evaluates multimodal LLMs on sequential clinical decision-making in oncology. Two cohorts: HANCOCK (multimodal pathology — TMA images, H&E slides, cell density) and MSK-CHORD (longitudinal genomics — genomic profiles, clinical timelines). Models act as agentic AI physicians, optionally using external tools (DrugBank, CONCH, PubMed). Each case contains multiple multi-choice questions; correctness is binary (matched vs. ground truth).

items
573items
AI subjects
25AI subjects
G

Multi-MoE

Released multiple-choice correctness results and ungraded MT-Bench conversations from the Multi-MoE serving study.

items
5,140items
AI subjects
11AI subjects
E

Multimodal STEM Assessments (Human-AI)

Released model responses and original fractional grades on university STEM assessment questions with archived input images.

items
249items
AI subjects
17AI subjects
M

NAEP LLM-as-Student

Recorded answer choices and correctness from the NAEP LLM-as-student study, linked to original NAEP question pages.

items
488items
AI subjects
11AI subjects
M

NaturalReasoning

The complete released NaturalReasoning subset: synthetic reasoning questions, available extracted references, and recorded Llama-3.3-70B-Instruct answers without correctness grades.

items
1,145,824items
AI subject
1AI subject
H

OCRBench v2

Bilingual visual text recognition, localization, parsing and reasoning with 10,000 released InternVL2.5-26B predictions across 30 task types.

items
9,868items
AI subject
1AI subject
S

OOD-Prediction

OOD-Prediction correctness dataset (Huang et al., ICML 2025): released prompts for 5 tasks (IOI, PriceTag, RAVEL, MMLU, Unlearn Harry Potter) under in-distribution and OOD settings, each prompt labelled correct/wrong by the target model. subject = target model; item = the prompt; response = 1 correct / 0 wrong.

items
142,805items
AI subjects
2AI subjects
E

OS-Harm

Released computer-use executions with separately identified safety and task-completion assessments.

items
512items
AI subjects
7AI subjects
EI

PeruMedQA

Spanish medical multiple-choice questions from Peruvian residency examinations, with released outputs from eleven medical model configurations.

items
8,259items
AI subjects
11AI subjects

PhyBlock

Visual 3D block assembly planning with released model outputs, scene images, candidate blocks and dependency annotations. This import covers the assembly branch, not aggregate paper rows or VQA references without predictions.

items
1,362items
AI subjects
21AI subjects
TH

ProofAug

ProofAug: cumulative theorem-proving results of deepseek-math-7b-base + ProofAug (mixed prompting strategy, <=2100 queries/problem, Isabelle) on the curated miniF2F-test — a full 244-problem binary solved/unsolved matrix (161 solved, 66.0%).

items
244items
AI subject
1AI subject
UN

Psychosis-bench

Released turn-level ratings of delusion confirmation, harm enablement and safety intervention in scripted conversations with eight models.

items
2,688items
AI subjects
8AI subjects
T

Pxplore

Recorded next-lesson recommendations from fourteen Pxplore policy configurations, with per-criterion learner-state alignment judgments for fifty released sessions.

items
787items
AI subjects
14AI subjects
TM

REAL

REAL tests whether web agents can complete user tasks on deterministic clones of real websites, counting an attempt as successful only when every one of the task's automatic checks passes.

items
233items
AI subjects
33AI subjects
P

RefGrader

Recorded grading of Olympiad mathematical proofs by reference-aided language-model workflows.

items
3,597items
AI subjects
31AI subjects
S

ResearchCodeBench

ResearchCodeBench tests whether models can implement a missing piece of a recent machine learning paper's code, given the paper itself and the surrounding file, with curated tests deciding whether the filled in code works.

items
212items
AI subjects
31AI subjects

RISEBench

Reasoning-informed visual editing on 360 temporal, causal, spatial and logical tasks. This import contains 21 author-released model exports and two historical BAGEL configurations, with individual image-completion judgments and separate grading protocols.

items
1,080items
AI subjects
23AI subjects

Scale MRT (Monitor Red Teaming)

Native final suspicion assessments made by monitor models on released SHADE-Arena and CUA-SHADE-Arena agent trajectories.

items
34,698items
AI subjects
633AI subjects
Y

SciArena

Recorded human preferences between scientific literature-grounded model responses, with their released questions and retrieved paper banks.

items
8,393items
AI subjects
23AI subjects

SciGym

Iterative scientific-discovery trajectories on 137 systems-biology tasks. Agents receive a partial SBML model, design simulated experiments, and submit a reconstructed model. The curated release includes six reaction-recovery metrics derived from final models and explicitly unavailable trajectory-error grades.

items
959items
AI subjects
6AI subjects

SVRPBench

Stochastic vehicle routing benchmark. This curation preserves the released individual route exports and their cost, constraint violation, feasibility, runtime, and robustness metrics.

items
40items
AI subjects
4AI subjects
SP

SWE-smith

Released SWE-agent run labels and formatted training conversations on synthetic SWE-smith software tasks.

items
15,616items
AI subjects
3AI subjects

SWE-bench-Live

Evaluates coding agents on resolving real repository issues using per-instance test outcomes.

items
2,202items
AI subjects
65AI subjects

SWE-PolyBench

Evaluates coding-agent issue resolution using repository tests.

items
547items
AI subjects
7AI subjects

TabArena

Individual test-row predictions from eight published TabArena methods, with derived classification correctness or regression squared error.

items
346,195items
AI subjects
384AI subjects
S

tau2-bench

Measures tool-using customer-service agents in dual-control conversations.

items
2,388items
AI subjects
31AI subjects

3EED

Released 3EED baseline predictions for outdoor language-conditioned 3D grounding, with original image and LiDAR inputs and native box-overlap grades.

items
8,616items
AI subjects
4AI subjects

TUMLU

Released Turkic-language multiple-choice model attempts with their complete recorded few-shot context, shuffled choices and original answer-extraction protocol.

items
133,944items
AI subjects
30AI subjects
VU

WikiHowAgent

Released completion outcomes for multi-model teacher–learner conversations grounded in WikiHow tutorials.

items
14,287items
AI subjects
12AI subjects

2024

61
UM

AdvPrompter

AISafetyLab evaluation of 50 AdvPrompter prompts against Vicuna-7B-v1.5, without a defense and with SafeDecoding. Each released judgment records whether Llama Guard 3 classified the response as unsafe.

items
50items
AI subjects
2AI subjects
IG

AfriMed-QA

Released model responses to multiple-choice medical questions in African healthcare contexts.

items
42,494items
AI subjects
34AI subjects
E

AgentDojo

Evaluates task utility and attacker-goal success for tool-using agents exposed to prompt injection.

items
1,081items
AI subjects
29AI subjects
SA

AppWorld

Evaluates interactive coding agents on tasks spanning a simulated collection of applications.

items
585items
AI subjects
18AI subjects
AM

BABILong

Released language-model answers to bAbI reasoning questions embedded in long distractor passages.

items
526,202items
AI subjects
83AI subjects
U

BertaQA

Released API-model responses to parallel English and Basque trivia questions about local Basque culture and global topics.

items
51,760items
AI subjects
6AI subjects

Bridging-the-Gap African Languages

Published multilingual question-answering results on Winogrande, Belebele and three clinical MMLU sections, with base and fine-tuned models.

items
132,418items
AI subjects
417AI subjects
MM

Causal Axioms (Axiomatic Training)

Axiomatic Training causal-reasoning evaluation: per-item Yes/No d-separation questions over linear causal chains of length 7-15 (500 items per length), answered by the baseline Meta-Llama-3-8B-Instruct and its axiomatic-training LoRA finetune. Response is binary (1 = prediction matches the gold Yes/No label).

items
4,500items
AI subjects
2AI subjects
S

ClashEval

ClashEval records how six language models answer factual questions with and without altered retrieved context. The import retains each released context-conditioned attempt and one reused closed-book answer per model/question.

items
12,495items
AI subjects
6AI subjects
TU

ComplexBench

Released model generations for complex Chinese instructions with composed grading requirements; historical judgments are unavailable.

items
1,150items
AI subjects
15AI subjects

CORE-Bench

Evaluates agents on reproducing scientific code and answering questions about its outputs.

items
270items
AI subjects
11AI subjects
S

CROW

Historical model outputs and keyword-based attack-success grades from CROW development exports.

items
1,984items
AI subjects
34AI subjects

CRUXEval

Recorded execution verdicts and postprocessed generations for input and output prediction on Python functions.

items
4,000items
AI subjects
31AI subjects

Data-Juicer 2.0

Published processing-time measurements for Data-Juicer 2.0 deduplication, CPU/GPU operators and cloud-scale workloads.

items
12items
AI subjects
11AI subjects

Embodied Agent Interface

Released BEHAVIOR goal-interpretation answers: full task prompts, symbolic goal references and recorded LLM outputs; historical grades are unavailable.

items
100items
AI subjects
25AI subjects
KM

ERBench

Answer correctness in ERBench's released binary-question CSVs across eight model folders and seven domain/task splits.

items
32,163items
AI subjects
8AI subjects
M

Few-Shot TTT (BIG-Bench Hard)

Released five-example previews for BIG-Bench Hard evaluations of Llama-3.1-8B-Instruct under eleven zero-shot, few-shot ICL and test-time training interventions. These previews cover 7,425 observations, not the complete evaluation sets used to calculate the accompanying task accuracies.

items
1,310items
AI subjects
11AI subjects

GoodAI LTM Benchmark

Released long-term-memory tests for conversational agents, preserving task definitions, scheduling conditions and complete recorded conversations.

items
452items
AI subjects
37AI subjects
UC

HarmBench

Released model generations and native attack-success judgments from the HarmBench text and multimodal playground.

items
62,160items
AI subjects
33AI subjects
U

AIR-Bench 2024

AIR-Bench tests model replies against a taxonomy of risks drawn from government regulations and company policies. This import preserves the published refusal/safety grades, including intermediate values.

items
7,723items
AI subjects
66AI subjects
UC

HarmBench

HarmBench measures whether a model refuses harmful requests, covering behaviors such as cybercrime, misinformation and chemical or biological harm, with a classifier judging each reply as safe or unsafe. Recorded scores retain intermediate judge ratings; 1 indicates the best rubric score and 0 the worst.

items
403items
AI subjects
81AI subjects

Image2Struct

Original LaTeX and music-sheet image-to-code predictions with their recorded multimodal requests and per-metric grades. Each response is one published grading dimension of an attempt, not an independent model generation. Webpage exports redact their stimuli and outputs and remain raw-only.

items
30,636items
AI subjects
14AI subjects

InfiBench

Per-attempt programming-question grades from the author-released InfiBench evaluation logs, including three human-answer comparators and the historical two-model development release.

items
369items
AI subjects
121AI subjects

KMMLU

Published Korean multiple-choice predictions from the independently maintained Korean LLM Benchmark Suite, matched to the original KMMLU question bank.

items
35,015items
AI subjects
30AI subjects

KorMedMCQA

Published Korean healthcare licensing-exam predictions from the independently maintained Korean LLM Benchmark Suite.

items
3,009items
AI subjects
7AI subjects
T

LexEval

Released Chinese legal predictions across 23 tasks and two prompting settings, using native accuracy or ROUGE-L without thresholding.

items
39,908items
AI subjects
38AI subjects
U

LINGOLY

Released model answers to individual LINGOLY question parts, retaining complete recorded prompts and full-context/no-context conditions.

items
12,463items
AI subjects
11AI subjects

LiveBench

Published per-question grading events from the LiveBench release history across six task categories.

items
1,286items
AI subjects
200AI subjects
T

LLM-Uncertainty-Bench

Released six-option logits for 25 model labels on five adapted multiple-choice datasets, with three prompting strategies and base/chat input formats.

items
149,793items
AI subjects
102AI subjects
S

MedHELM Medbullets

MedHELM medbullets: per-(model, item) exact_match in {0,1} (1=correct) on 308 USMLE Step 2/3-style clinical vignette multiple-choice questions. 13 models from the MedHELM v4.0.0 release.

items
298items
AI subjects
13AI subjects
S

MedHELM MedCalc-Bench

MedHELM medcalc_bench: per-(model, item) medcalc_bench_accuracy in {0,1} (1=correct) on 1000 clinical calculations from a patient note (risk scores, dosing, lab-derived values). 13 models from the MedHELM v4.0.0 release.

items
992items
AI subjects
13AI subjects
S

MedHELM MEDEC

MedHELM medec: per-(model, item) medec_error_flag_accuracy in {0,1} (1=correct) on 597 detecting whether a clinical note contains a medical error (error-flag accuracy). 13 models from the MedHELM v4.0.0 release.

items
597items
AI subjects
13AI subjects
UU

MJ-Bench released judge observations

Original released multimodal judge preferences and bias ratings, linked to unchanged MJ-Bench image inputs.

items
26,564items
AI subjects
82AI subjects

MMedBench (MedS-Ins rationale examples)

All 300 released MedS-Ins rationale-generation examples on MMedBench: 50 attempts in each of six languages, with full recorded requests, reference explanations and generated outputs. The release has no saved grades.

items
300items
AI subject
1AI subject
U

MMLU-Pro

Released model predictions on multiple-choice questions across fourteen academic domains.

items
14,226items
AI subjects
48AI subjects

MOCHI

Published per-image-set accuracy estimates for vision models identifying the different object among multiple views.

items
3,717items
AI subjects
24AI subjects
U

MoE-CAP

Released requests, outputs and correctness results for mixture-of-experts models under different serving configurations.

items
10,065items
AI subjects
50AI subjects
S

Large Language Monkeys: Monkey Business

Complete released prompts, model-generated solutions and per-sample correctness flags from the original Monkey Business repeated-sampling collection.

items
525items
AI subjects
13AI subjects
H

NESTER

NESTER: dataflow-guided neuro-symbolic type inference for Python. Released per-item data covers the human evaluation of high-level program generation on ManyTypes4Py samples: GPT-4, GPT-4o and GPT-4o mini generations each graded on four binary criteria (correctness, explainability, factuality, consistency) by human annotators. Recovered from deleted xlsx blobs in the authors' GitHub repository.

items
356items
AI subjects
3AI subjects
N

NYU CTF Bench

Published challenge-level flag-recovery assessments from 15 NYU CTF leaderboard submissions, with original challenge inputs and released agent trajectories.

items
400items
AI subjects
15AI subjects
P

PKU-SafeRLHF

PKU-SafeRLHF per-response safety ratings for Alpaca-family models.

items
39,994items
AI subjects
3AI subjects

PRISM Alignment

Human ratings and selected-response flags for 21 generating models in PRISM conversations.

items
53,926items
AI subjects
21AI subjects
S

ProX Math CPT Evals

ProX (Programming Every Example) pre-training data refinement — released per-item math evals. Coverage is the partial slice the authors released: TinyLlama-1.1B continually pre-trained on the original OpenWebMath corpus, at two intermediate checkpoints (0.5B and 1B training tokens), scored per-item (binary correctness) on the paper's 9-task few-shot CoT math suite (GSM8K, MATH, SVAMP, ASDiv, MAWPS, TabMWP, MathQA, MMLU-STEM, SAT-Math). Per-item files for the ProX-refined CPT counterparts were not released.

items
12,052items
AI subjects
2AI subjects
S

Putnam-AXIOM

University-level Putnam competition math problems; response is a human-graded reasoning rubric (5 ordinal dimensions) on each model solution.

items
2,500items
AI subjects
2AI subjects
HU

REVOLVE

REVOLVE solution-optimization evaluation: Llama-3.1-8B-Instruct performs test-time training on GPQA-Diamond (198 items) and MMLU Machine Learning (112 items) multiple-choice questions, optimized either by the TextGrad baseline (v1) or by REVOLVE (v2, response-evolution tracking). Binary exact-match correctness of the extracted answer letter at each optimization stage (initial zero-shot, iterations 1-3, majority-vote final).

items
310items
AI subjects
2AI subjects

SafeAgentBench (TRIDENT instruction decisions)

Released TRIDENT instruction-rejection observations on the SafeAgentBench task bank. Six system configurations report whether an instruction is blocked or refused; this subset measures system behavior, not embodied task-completion success.

items
665items
AI subjects
6AI subjects
H

StatQA

Statistical method and column selection over described tabular datasets. Preserve the released general-analysis answers and qualitative explanatory replies.

items
1,183items
AI subjects
46AI subjects
G

SubjECTive-QA

Classify six subjective dimensions of earnings-call question/answer pairs against released human labels.

items
10,921items
AI subjects
5AI subjects
E

SynthPAI

SynthPAI: personal-attribute inference from synthetic Reddit profiles. 294 synthetic author profiles (7823 comments); 18 LLMs infer up to 8 personal attributes (age, sex, location, birthplace, education, occupation, income, relationship status) from comment history. Each model gives a ranked guess list per attribute, graded 0/0.5/1 against human-curated ground truth.

items
9,220items
AI subjects
18AI subjects

tau-bench

Measures customer-service tool-using agents with simulated users.

items
201items
AI subjects
49AI subjects

TheAgentCompany

Evaluates agents on workplace tasks in a simulated company using task-specific checkpoint scores.

items
175items
AI subjects
19AI subjects

VHELM

Original VHELM vision-language requests, image inputs, recorded outputs and release-declared task grades.

items
93,127items
AI subjects
112AI subjects
BB

Visual Riddles

Visual Riddles: a commonsense and world-knowledge challenge for large vision and language models. Each item is an AI-generated image with an open-ended question requiring visual reasoning plus world knowledge. Per-(model, item) responses are human-rated correct/incorrect on the open-ended task.

items
793items
AI subjects
8AI subjects

VisualWebArena

Released automatic and human success judgments of multimodal web-agent trajectories on VisualWebArena.

items
200items
AI subjects
5AI subjects

VL-RewardBench

VL-RewardBench: judge VLM accuracy on (image, query, chosen, rejected) preference pairs.

items
1,532items
AI subjects
16AI subjects
UU

WildVision-Bench

Image-based user requests with model answers and GPT-4o preference judgments against a fixed Claude-3-Sonnet comparison answer.

items
500items
AI subjects
22AI subjects

WONDERBREAD

Published workflow question answers, demonstration judgments, SOP rankings, SOP generation and per-screenshot segmentation outcomes. Original SOPs, action traces and screenshots supply the actual task context.

items
9,091items
AI subjects
38AI subjects

WorkArena

Human assessments of released GenericAgent trajectories on ServiceNow tasks from WorkArena and WorkArena++.

items
385items
AI subjects
8AI subjects

2023

46

BeaverTails

Released safety judgments of 560 outputs from four language models on 140 red-team prompts.

items
420items
AI subjects
4AI subjects
UA

BIRD (text-to-SQL)

Published Petavue evaluations of text-to-SQL models on a 360-question BIRD development subset.

items
360items
AI subjects
120AI subjects
ST

C-Eval

Chinese multiple-choice question bank with released item-level option predictions from the Contamination Detector study.

items
13,951items
AI subjects
11AI subjects

ChatGPT Drift (LLMDrift)

Recorded 2023 GPT-3.5/GPT-4 snapshots across eight tasks, retaining complete outputs and original grading semantics.

items
11,479items
AI subjects
12AI subjects
M

CMMLU

Chinese multiple-choice knowledge questions with complete OpenCompass model predictions and a versioned reconstruction of option-extraction accuracy.

items
23,160items
AI subjects
22AI subjects
U

DecodingTrust

Released model outputs from six DecodingTrust perspectives, with complete source context and explicit missing grades.

items
437,586items
AI subjects
11AI subjects
LM

Do-Not-Answer

Do-Not-Answer baseline-refusal benchmark: 939 refusal-worthy prompts asked directly (no adversarial wrapper) of 6 LLMs (GPT-4, GPT-3.5, Claude, ChatGLM2, Llama-2-7B-chat, Vicuna-7B), with two annotation labels per cell: binary harmfulness and a six-category action label.

items
1,876items
AI subjects
6AI subjects
U

DORIS-MAE

Published Anno-GPT relevance judgments and explanations for scientific requirement–abstract pairs.

items
165,104items
AI subject
1AI subject
U

EgoSchema (released LLoVi configurations)

Published LLoVi question-answering results on the 500-question public EgoSchema subset, retaining the available caption-based inputs and eleven model/prompt configurations.

items
4,500items
AI subjects
11AI subjects
WN

EVOUNA

EVOUNA (QA-Eval): per-(model, question) human correctness judgments for five Open-QA systems (FiD, GPT-3.5/text-davinci-003, ChatGPT, GPT-4, New Bing) answering the same Open-domain questions from Natural Questions (3,610) and TriviaQA (2,000). Each model answer was manually judged correct/incorrect by a human annotator.

items
5,609items
AI subjects
5AI subjects

FETV

Released video-generation measurements for four models on 619 prompts, including individual human ratings and five automatic alignment metrics.

items
20,788items
AI subjects
4AI subjects

FinanceBench

Evaluates whether models can answer financial questions about public-company filings across closed-book and context-assisted settings.

items
150items
AI subjects
16AI subjects

FIND

Released function-interpretation conversations and available native grades for numeric, string and synthetic-neuron functions.

items
10,397items
AI subjects
5AI subjects
U

GHOSTS

Human ratings of ChatGPT and GPT-4 answers to natural-language mathematics questions with released prompt text.

items
478items
AI subjects
3AI subjects
U

HallusionBench

Released image-context reasoning outputs from OpenVLMRecords, with original HallusionBench sample outputs preserved separately.

items
1,205items
AI subjects
254AI subjects

HELM CLEVA (Chinese)

Chinese language evaluation across 21 released task configurations and four model exports. Preserve the declared task metric: item grades where published, and null grades for attempts evaluated only by corpus-level metrics.

items
7,536items
AI subjects
30AI subjects
P

SimpleSafetyTests

SimpleSafetyTests checks whether a model responds safely to blunt requests touching severe harms such as self harm, violence and illegal activity, with a classifier marking each reply safe or unsafe. Recorded scores retain intermediate judge ratings; 1 indicates the best rubric score and 0 the worst.

items
100items
AI subjects
81AI subjects
S

HELM Thai Exam

HELM evaluation on five Thai standardized examination sets: A-Level, IC, ONET, TGAT and TPAT1. Preserve native multiple-choice prompts and exact-match grades.

items
565items
AI subjects
42AI subjects
BU

XSTest

XSTest checks whether models refuse harmless prompts that merely sound dangerous, pairing them with genuinely unsafe lookalikes so that both excessive caution and unsafe compliance show up. Recorded scores retain intermediate judge ratings; 1 indicates the best rubric score and 0 the worst.

items
450items
AI subjects
81AI subjects

IgakuQA

Published 2018–2022 Japanese medical-exam answers from four LLM configurations and a separately identified human-majority-derived comparator.

items
2,942items
AI subjects
5AI subjects

InterCode

Released interactive coding episodes in SQL, Bash, Python and capture-the-flag environments, with their original maximum rewards and complete trajectories.

items
2,484items
AI subjects
26AI subjects
NA

LawBench

Released Chinese legal-task predictions with complete recorded prompts and deterministic native scoring.

items
233,168items
AI subjects
51AI subjects
S

LegalBench

Legal reasoning in five LegalBench tasks evaluated by the 155 released HELM Lite v1.0.0 runs; preserve original per-instance quasi-exact-match grades.

items
9,393items
AI subjects
31AI subjects

Lo-Hi Drug Discovery Benchmark

Complete published molecule-level prediction records for the Lo and Hi drug-discovery tasks, with their reference measurements and native aggregate ranking protocol. Individual grades are unavailable.

items
42,080items
AI subjects
213AI subjects
U

MathVista MINI

MathVista testmini visual-mathematical questions with released OpenVLMRecords predictions, cached answer extractions, original images and explicitly derived historical-comparator grades.

items
1,000items
AI subjects
263AI subjects
S

MedHELM EHRSQL

MedHELM ehr_sql: per-(model, item) ehr_sql_execution_accuracy in {0,1} (1=correct) on 1000 translating a natural-language clinical question into SQL over an EHR database. 13 models from the MedHELM v4.0.0 release.

items
1,000items
AI subjects
13AI subjects
S

MedHELM RaceBasedMed

MedHELM race_based_med: per-(model, item) exact_match in {0,1} (1=correct) on 167 detecting whether a medical answer relies on harmful race-based reasoning. 13 models from the MedHELM v4.0.0 release.

items
167items
AI subjects
13AI subjects

Mind2Web candidate generation

Recorded candidate-generation rankings from the Mind2Web authors' fine-tuned DeBERTa-v3-base cross-encoder, joined to complete released train and test action-step inputs.

items
17,153items
AI subject
1AI subject
S

MMBench V1.1 recorded answers

Recorded English MMBench V1.1 answers, original image/question inputs and explicit deterministic per-response grading.

items
12,175items
AI subjects
251AI subjects
IU

MMMU_DEV_VAL

College-level multimodal questions across 30 subjects, with per-question model predictions released by OpenVLMRecords.

items
1,050items
AI subjects
253AI subjects

NIS3D

Recorded 3D nuclei segmentation quality on six embryonic microscopy volumes.

items
30items
AI subjects
6AI subjects

PRM800K

Released human process ratings of a fixed GPT-4-derived research generator on MATH problems.

items
973,120items
AI subject
1AI subject

ProteinInvBench

Original ProteinInvBench CATH4.2 and CATH4.3 inverse-folding predictions, matched to the released backbone coordinates and recorded model configurations.

items
7,410items
AI subjects
16AI subjects

QATCH

Original QATCH question-answering and semantic-parsing results, with complete table contexts and five distinct recorded grading protocols.

items
36,458items
AI subjects
6AI subjects

Rakuda

Rakuda Japanese open-ended questions, with Shaberi model answers and separately recorded LLM-judge ratings.

items
280items
AI subjects
647AI subjects

SIB-200

Original topic-classification outputs in 205 languages, with recorded language-specific classifier runs and zero-shot LLM replies.

items
83,640items
AI subjects
3,343AI subjects

Stanford-ORB

Original capture-level assessments of 3D object inverse-rendering methods on measured real-world images and geometry.

items
630items
AI subjects
13AI subjects
U

SugarCrepe

SugarCrepe: a non-gameable vision-language compositionality benchmark of ~7.5k (image, positive caption, hard-negative caption) triples across 7 hard-negative types (add/replace/swap of attributes, objects, relations). Models must pick the caption that matches the COCO-2017 image. This build ingests the released per-item GPT-4V responses from the official repo (the only model with per-instance outputs released; the 17 CLIP models in the paper have only aggregate accuracy), in two caption-presentation orders (positive-first / negative-first).

items
15,024items
AI subject
1AI subject

Tensor Trust

Released game records of model access-granting behavior under human-written defense prompts and inputs.

items
570,791items
AI subjects
3AI subjects
A

Tulu Human Evaluation

Measures human acceptability of saved instruction-following model responses.

items
1,028items
AI subjects
4AI subjects
T

UltraFeedback

Measures four aspects of released model completions using recorded GPT-4 feedback.

items
255,820items
AI subjects
743AI subjects
HU

VisIT-Bench

Measures instruction following through original human and GPT-4 judgments, retaining the original visual or caption inputs.

items
2,391items
AI subjects
19AI subjects

XSTest

Measures compliance and refusal on safe prompts and unsafe contrasts.

items
2,250items
AI subjects
5AI subjects

Earlier (pre-2023)

42
A

AI2D_TEST

Multiple-choice science-diagram questions, with primary per-question model predictions released by OpenVLMRecords and deterministically derived exact-match grades.

items
3,088items
AI subjects
254AI subjects

Aider C++ prompting study

Native Aider polyglot results from Parasoft's MISRA prompting study: 5,356 attempts on 26 C++ tasks using seven model labels and sixteen named experimental arms. One GPT-5 arm changes its instruction text between runs, yielding seventeen recorded agent configurations.

items
26items
AI subjects
17AI subjects
G

ARC-AGI

ARC Prize's released model attempts on the public evaluation tasks of ARC-AGI-1 and ARC-AGI-2, which test the inference of transformations from example grids.

items
526items
AI subjects
80AI subjects
H

AtmosSci-Bench

Recorded language-model attempts on multiple-choice and open-ended atmospheric-science problems.

items
2,802items
AI subjects
54AI subjects

BBQ

Multiple-choice questions testing the effects of social biases in ambiguous and disambiguated contexts.

items
204,201items
AI subjects
7AI subjects

Decompile-Bench: released IDA and Ghidra outputs

Released decompiler outputs for individual C/C++ functions from HumanEval, MBPP and GitHub2025. Derive the authors' edit-similarity metric deterministically from the original outputs and reference source. This metric measures textual similarity, not executable correctness.

items
72,059items
AI subjects
2AI subjects

DevBench

Published model choice scores for lexical mapping, visual vocabulary and grammatical comprehension, with available complete image inputs.

items
271items
AI subjects
19AI subjects

DPAI Arena

Published blind and test-informed software-agent evaluations on enterprise Java/Spring tasks.

items
306items
AI subjects
26AI subjects

DQVis expert-reviewed generations

Published expert judgments of generated natural-language queries and associated biomedical visualization specifications.

items
356items
AI subject
1AI subject
VU

DTap-Bench

Released task-utility and attacker-success verdicts for ten agent-framework/model configurations, with complete related native histories.

items
13,392items
AI subjects
10AI subjects
ZE

EduGuardBench

Published teaching-question answers and adversarial safety judgments for fourteen simulated-teacher model configurations.

items
7,494items
AI subjects
14AI subjects
U

EDUMATH

Released mathematical word-problem generations and two published automatic quality judgments for eleven generator models.

items
76items
AI subjects
11AI subjects

EngDesign

Engineering-design tasks with released simulation-based pass/fail judgments. The native release includes single-answer trials and iterative refinement chains.

items
101items
AI subjects
16AI subjects
S

HAIID

Human judgments before and after advice on five classification tasks, with recorded ResNet-18 advice for the dermatology task.

items
152items
AI subjects
1,126AI subjects

HELM African MMLU + Winogrande

HELM evaluation of clinical MMLU and Winogrande in 11 African languages, preserving published per-request exact-match grades.

items
72,223items
AI subjects
22AI subjects
A

Anthropic Red Team

Anthropic Red Team tests whether models avoid complying with adversarial prompts taken from human red teaming conversations, with a safety classifier judging each response as safe or unsafe. Recorded scores retain intermediate judge ratings; 1 indicates the best rubric score and 0 the worst.

items
995items
AI subjects
81AI subjects
N

BBQ

BBQ tests whether models pick the correct answer to multiple choice questions about people from different social groups, asking each question once with too little context to answer it and once with the context that settles it.

items
2,000items
AI subjects
81AI subjects
AU

BOLD

BOLD measures biases in open-ended language generation. This import preserves HELM's published fraction of toxic completions for each request, with lower values indicating less toxicity.

items
2,028items
AI subjects
42AI subjects
U

RealToxicityPrompts

RealToxicityPrompts measures how often a model continues a sentence fragment with text a toxicity classifier flags as toxic, using prompts drawn from both toxic and innocuous starting text.

items
2,006items
AI subjects
42AI subjects
US

IgakuQA119

Released Qwen2.5-72B-Instruct answers on all 400 questions of the 119th Japanese Medical Licensing Examination, with original explanations and source image resources.

items
400items
AI subject
1AI subject

Indeterminacy Experiments

Native forced-choice and response-set judgments from nine LLM raters on eleven published tasks, with complete prompt and input-order associations.

items
4,284items
AI subjects
9AI subjects
SU

IneqMath

Olympiad inequality problems expressed as bound estimation or relation prediction. This release preserves 1000 distinct GPT-4o-mini example outputs across four prompt conditions, with individual grades unavailable.

items
1,000items
AI subject
1AI subject
UE

JailbreakBench

Released final target-model outputs under five jailbreak methods, with primary and available historical judge decisions kept as separate grading protocols.

items
2,001items
AI subjects
10AI subjects
S

JETTS generator response pool

Original scored generator responses released by JETTS across eight math, code and instruction-following component datasets.

items
5,940items
AI subjects
8AI subjects

Katakomba

Released final-checkpoint NetHack episode outcomes for 572 independently trained offline-RL policies across 38 character-configuration tasks.

items
38items
AI subjects
572AI subjects
S

MedHELM HEAD-QA

MedHELM head_qa: per-(model, item) exact_match in {0,1} (1=correct) on 1000 Spanish healthcare specialization (MIR/medicine/pharmacy/nursing) exam questions. 13 models from the MedHELM v4.0.0 release.

items
1,000items
AI subjects
13AI subjects
S

MedHELM MedMCQA

MedHELM med_mcqa: per-(model, item) exact_match in {0,1} (1=correct) on 1000 Indian medical-entrance (AIIMS/NEET-PG) multiple-choice questions. 13 models from the MedHELM v4.0.0 release.

items
1,000items
AI subjects
13AI subjects
S

MedHELM MedQA

MedHELM med_qa: per-(model, item) exact_match in {0,1} (1=correct) on 1000 US medical licensing exam (USMLE) multiple-choice clinical questions. 13 models from the MedHELM v4.0.0 release.

items
1,000items
AI subjects
13AI subjects
S

MedHELM PubMedQA

MedHELM pubmed_qa: per-(model, item) exact_match in {0,1} (1=correct) on 1000 biomedical research questions answered yes/no/maybe over PubMed abstracts. 13 models from the MedHELM v4.0.0 release.

items
1,500items
AI subjects
13AI subjects

MLR-Bench

Released idea-generation and research-proposal reviews for 201 ML research tasks, six generator models and two LLM reviewers. Each observation is one rubric dimension of a released review.

items
5,226items
AI subjects
6AI subjects
UC

MMLU

Measures academic and professional knowledge through multiple-choice likelihood scoring.

items
14,015items
AI subjects
150AI subjects

NanoBaseLib RNA Modification Detection

Released site-level predictions for the m6A and m5C RNA-modification tasks, with the original reference assays and aggregate evaluation protocol.

items
45,970items
AI subjects
10AI subjects
A

PlanBench

Measures whether language models produce valid plans in classical planning domains.

items
11,076items
AI subjects
42AI subjects

RADAR

Recorded code-agent example from the RADAR benchmark for reasoning over imperfect tables.

item
1item
AI subject
1AI subject

RClicks: released clickability-model measurements

Per-stimulus click-distribution measurements for 16 released clickability algorithms and variants across five image datasets. Retain original images, object masks, previous segmentation masks and human click references.

items
29,035items
AI subjects
16AI subjects
TT

RealTime QA

Released weekly news-question predictions in multiple-choice and free-text generation settings.

items
7,954items
AI subjects
94AI subjects

Reasoning Gym: original zero-shot completions

Individual completions and their original algorithmic rewards from the released easy and hard zero-shot experiments. Retain repeated attempts, partial credit and complete prompts and outputs.

items
56,181items
AI subjects
14AI subjects

SummEval

Human assessments of model-generated CNN/DailyMail summaries along four quality dimensions.

items
800items
AI subjects
16AI subjects
UO

TruthfulQA-MC

Measures truthfulness by the likelihood assigned to correct and misleading answers.

items
817items
AI subjects
150AI subjects

WMT MQM: per-rater error indicators

Published expert translation-error annotations represented by the existing six-category, three-severity binary readout.

items
207,090items
AI subjects
113AI subjects

Meta-analyses

Coverage and timelines use the full data bank. The score heatmap follows benchmark type, not the current search or filters.

Provider CoverageAll 270

Providers ranked by the distinct benchmark items evaluated for their most-measured AI subject.

#AI subjectItems asked
  1. 1Meta

    Meta

    Llama 3.3 70B

    1,212,405
  2. 2OpenAI

    OpenAI

    GPT-3.5

    967,031
  3. 3Mistral

    Mistral

    Mistral 7B

    242,408
  4. 4Google

    Google

    Gemini 1.5 Pro

    158,234
  5. 5Anthropic

    Anthropic

    Claude 3.5 Sonnet

    137,960
  6. 6Microsoft

    Microsoft

    Phi-3 Medium

    126,133
  7. 7DeepSeek

    DeepSeek

    V3

    110,776
  8. 8S

    Stanford

    Alpaca 7B

    100,174
  9. 9Alibaba

    Alibaba

    Qwen2.5 72B

    90,051
  10. 10Cohere

    Cohere

    Command R

    75,256
  11. 11T

    TII

    Falcon 40B

    69,109
  12. 12L

    LMSYS

    Vicuna 33B

    67,813
Showing top 8 of 12
Domain CoverageAll 270

Domains ranked by evaluated items. Multi-domain benchmarks count toward every applicable domain, so totals overlap.

#DomainItems
  1. 1

    Reasoning

    76 benchmarks

    2,669,114
  2. 2

    Mathematics

    30 benchmarks

    2,256,242
  3. 3

    Safety

    42 benchmarks

    1,681,425
  4. 4

    Science

    25 benchmarks

    1,396,546
  5. 5

    General

    49 benchmarks

    1,087,623
  6. 6

    Reward Modeling

    4 benchmarks

    1,005,261
  7. 7

    Multilingual

    17 benchmarks

    971,180
  8. 8

    Knowledge

    21 benchmarks

    514,199
  9. 9

    ML Engineering

    7 benchmarks

    353,930
  10. 10

    Preference

    10 benchmarks

    351,314
  11. 11

    Law

    5 benchmarks

    283,529
  12. 12

    Agents & Tool Use

    29 benchmarks

    250,186
  13. 13

    Cybersecurity

    4 benchmarks

    222,820
  14. 14

    NLP Tasks

    6 benchmarks

    215,280
  15. 15

    Medicine

    35 benchmarks

    135,906
  16. 16

    Finance

    6 benchmarks

    131,492
  17. 17

    Software Engineering

    24 benchmarks

    111,841
  18. 18

    Education

    14 benchmarks

    86,048
  19. 19

    Cultural

    2 benchmarks

    52,557
Showing top 8 of 19
Benchmark Release TimelineAll 270

Month-dated benchmarks, stacked by release month and colored by primary domain.

SafetyCybersecurityMedicineLawFinanceMathematicsSoftware EngineeringML EngineeringAgents & Tool UseScienceMultilingualEducationPreferenceReward ModelingNLP TasksKnowledgeReasoningGeneral

Showing recent releases. Swipe for earlier years.

2016 · 1
2017 · 0
2018 · 0
2019 · 3
2020 · 4
2021 · 7
2022 · 4
2023 · 47
2024 · 65
2025 · 80
2026 · 35

Not shown (no month-dated release): Aider C++ prompting study, AtmosSci-Bench, Decompile-Bench: released IDA and Ghidra outputs, DevBench, DPAI Arena, DQVis expert-reviewed generations, DTap-Bench, EduGuardBench, EDUMATH, EngDesign, HELM African MMLU + Winogrande, IgakuQA119, Indeterminacy Experiments, IneqMath, JailbreakBench, JETTS generator response pool, JudgeTuning LMSys human agreement, Katakomba, MLR-Bench, NanoBaseLib RNA Modification Detection, RADAR, RClicks: released clickability-model measurements, Reasoning Gym: original zero-shot completions, Tengu-Bench.

AI Subject Release TimelineAll 270

Dated AI subjects evaluated in the data bank, stacked by release month and colored by provider.

MetaOpenAIMistralGoogleAnthropicMicrosoftDeepSeekStanfordAlibabaCohereTIILMSYSOther

Showing recent releases. Swipe for earlier years.

2016 · 0
2017 · 0
2018 · 0
2019 · 2
2020 · 7
2021 · 9
2022 · 19
2023 · 115
2024 · 274
2025 · 184
2026 · 68

103 AI subjects with no recorded release date are not shown.

AI Subject by Domain ScoresAutomated

Mean binary score for each AI subject and domain. Multi-domain benchmarks contribute to every applicable column; hatching means no qualifying measurement.

Mean binary score0 – 100
Not measured

Swipe sideways for more domains.

AI subject
Showing top 15 of 660 AI subjects

Analysis package

Analyze the bank with torch_measure

A PyTorch-native toolkit for predictive evaluation of AI systems. The Data Bank supplies curated item-level evidence; torch_measure works downstream on compatible subject-by-item data for modeling and diagnostics.

Fit
Fit latent-variable models, including IRT and factor models, to compatible response tensors.
Diagnose
Study reliability, item fit, calibration, scalability, differential item functioning, and uncertainty.
Design
Build adaptive tests with Fisher-information, spanning, or random item-selection strategies.
Read the docsGitHubPyPI
Analysis quickstartPython 3.10+
pip install torch_measure
import torch
from torch_measure.models import Rasch

# Binary AI systems × items; NaN marks a missing response.
responses = torch.tensor(
    [
        [1, 0, 1, float("nan")],
        [0, 1, 1, 0],
        [1, 1, float("nan"), 1],
    ],
    dtype=torch.float32,
)

model = Rasch(
    n_subjects=responses.shape[0],
    n_items=responses.shape[1],
)
model.fit(responses, max_epochs=500, verbose=False)

print(model.ability.detach())
print(model.difficulty.detach())

This Rasch example is for binary responses. When preparing a compatible tensor from the Data Bank, retain the selected benchmark, trial, and test-condition context alongside it.

Definition of Data Schema

  • A subject is the complete configuration of an AI system under evaluation: the base model (typically a large language model) together with the setting that alters its behavior, including its access date, harness name, harness version, and reasoning effort. Anthropic's Demystifying evals for AI agents defines a harness (which is also called scaffold) as "the system that enables a model to act as an agent: it processes inputs, orchestrates tool calls, and returns results." Two evaluations of the same base model under different configurations (e.g., the same base model with two harnesses) can behave quite differently, constituting two different subjects.

  • An item (i.e., a question or a task) comprises the complete stimulus presented to the subject and its grading protocol. The stimulus includes the underlying prompt, attachments, and item features that alter its presentation (such as few-shot examples or prompt phrasing). The grading protocol comprises the grading criterion (reference answer, rule, or both) and the verifier that applies it. The same stimulus evaluated under different grading protocols has distinct item IDs; its text-only content_hash remains available for finding shared prompt text.

  • An interactor is any party other than the subject that participates in producing a response, such as the attacker probing the subject in an adversarial evaluation, the simulated user or human user the agent converses with, or the opponent subject in a pairwise comparison.

  • A test condition records a setting of the measurement occasion that varied while the model and the prompt content were held fixed, including the sampling temperature. The test condition is null when all information about the experiment is captured in other fields. We expect most information of the experiment to be documented in specific fields, and test conditions should be null most of the time.

  • A response is the grade of a trace for tuples of (subject, item, trial, interactor, test condition), typically on a ordinal scale with a small, finite set of categories. The typical case in AI evaluation is binary (with 1 represents correct or success, and 0 otherwise), but there is also the case where we have Likert scales. Continuous responses for the tuple (subject, item, trial, integrator, test condition) are empiricially rare, especially for frontier subjects.

Implementation of Data Schema

Each benchmark should have one folder called benchmarks/{dataset}/. The most important file in this folder is build.py, which is responsible for downloading the raw upstream data and adapt it to the data schema above. The downloaded data is stored in the raw/ and, in some cases, the docker/ folder. For a given upstream data source, the data curation done by the file build.py should be completely reproducible and well-documented. The build.py file relies on the metadata.yaml file, stored in the same directory, to provide metadata about the benchmark. The metadata.yaml together with shared dataset validation and benchmark-specific source checks under tests/benchmarks/, with reviewed expectations in characterization.yaml ensure the quality of the curation process.

The metadata.yaml file contains the declarative benchmark facts, such as benchmark name, expected number of item from benchmark technical report, expected number of subject, and trace coverage, as well as provider-owned source URLs and paper references. To see a complete list of fields and their descriptions in a metadata.yaml file, see benchmark_metadata_schema.yaml. We avoid duplicate data stored in this YAML file in other files to avoid drift. For the template of a file build.py, please see benchmarks/_template/build.py, which can be helpful when starting a new benchmark. Each build.py file is intentionally writen in a way that is readable and maintainable. The quality of the metadata.yaml file and characteristic test files are ensure by human reviewers.

A build.py file should generate at least 4 tables, saved as parquet files. For a complete schema of the table with detailed descriptions, please see parquet_schemas.yaml. These files are:

  • The file items.parquet lists the benchmark's items. Each row includes the exact item used to evaluate the subject and its required grading_criterion and verifier. The criterion is a JSON object with reference_answer and rule fields: each is a nonempty string or null, and at least one must be supplied. The verifier describes the grader or implementation that applies the criterion. Both contribute to item identity, together with the effective response scale inherited from the benchmark or specified in a mixed-scale criterion. Uniform scales are not duplicated in item records. Neither grading field is repeated in responses.parquet.

  • The file subjects.parquet lists the AI subjects the benchmark evaluated.

  • The file benchmarks.parquet is a single row of metadata about the benchmark itself. Its upstream data link is source_url. The derived has_reference_answer flag is true if any item has a reference answer; items graded solely by rules do not set this flag.

  • The file responses.parquet is a long-form table with one row per (subject, item, trial, test condition, interactors) tuple.

  • The optional file traces.parquet holds available interaction traces, with zero or one trace per response. Join it to responses.parquet on response_id; repeated attempts have distinct response IDs. The response table does not contain a trace placeholder. The trace file is omitted when no traces are available.

  • The file assets.parquet contains any additional assets needed for the benchmark, such as images or other files.

The scripts/build_measurement_tables/ folder holds shared Python helpers for registering subjects and items, deriving IDs, and validating and writing tables.

Citation

Cite this work

If you use the data we curated, please cite the following reference. The curation is released under CC BY-SA 4.0; individual benchmarks retain their upstream licenses.

BibTeX
@misc{measurementdb2026,
  title        = {The AI Measurement Data Bank},
  author       = {Truong, Nhi and Truong, Sang T. and Koyejo, Sanmi},
  year         = {2026},
  howpublished = {\url{https://aimslab.stanford.edu/measurement-db}},
  note         = {AIMS Lab, Stanford University}
}