Papers and preprints.
We believe progress in AI depends on our ability to measure it well. Our research develops the theory, methods, and infrastructure needed to make AI evaluation a rigorous science, spanning foundations, prediction, validity, incentives, and real-world impact.
Predictive evaluation
Modeling, scaling laws, and forecasting evaluation outcomes.
6 papers
A Measurement Science Roadmap: From Human Assessment to AI Evaluation
S Truong, N Goodman, E Brunskill, B Domingue, N Haber, S Koyejo
Preprint · 2026
Pretraining Scaling Laws for Generative Evaluations of Language Models
R Schaeffer, N Levi, B Miranda, S Koyejo
ICLR 2026 poster · 2025
Reliable and Efficient Amortized Model-based Evaluation
S Truong, Y Tu, P Liang, B Li, S Koyejo
ICML 2025 poster · 2025
Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks
R Schaeffer, PS Koura, B Tang, R Subramanian, AK Singh, T Mihaylov, et al.
arXiv preprint · 2025
How Do Large Language Monkeys Get Their Power (Laws)?
R Schaeffer, J Kazdan, J Hughes, J Juravsky, S Price, A Lynch, E Jones, et al.
ICML 2025 oral · 2025
Quantifying Variance in Evaluation Benchmarks
L Madaan, AK Singh, R Schaeffer, A Poulton, S Koyejo, P Stenetorp, et al.
arXiv preprint · 2024
Validity and reliability
Whether benchmarks measure what they claim, and how reliably, under bugs, saturation, and leakage.
4 papers
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
M Akhtar, A Reuel, P Soni, S Ahuja, PS Ammanamanchi, R Rawal, et al.
ICML 2026 main conference · 2026
Quantifying the Effect of Test Set Contamination on Generative Evaluations
R Schaeffer, J Kazdan, B Abbasi, KZ Liu, B Miranda, A Ahmed, F Berez, et al.
arXiv preprint · 2026
Fantastic Bugs and Where to Find Them in AI Benchmarks
S Truong, Y Tu, M Hardy, A Reuel, Z Tang, J Burapacheep, et al.
NeurIPS · 2025
Measurement to Meaning: A Validity-Centered Framework for AI Evaluation
O Salaudeen, A Reuel, A Ahmed, S Bedi, Z Robertson, S Sundar, et al.
arXiv preprint · 2025
Incentive, design, and governance
Incentive-aware evaluation, leaderboard design, and the governance of deployed AI.
6 papers
Strategic Evaluation: Incentivizing AI Capability Coverage with Private Benchmarks
S Truong, S Wang, N Haber, S Koyejo
Preprint · 2026
Public AI Benchmarks Are Broken, But Are Private Benchmarks the Answer?
S Truong, S Wang, A Wang, N Truong, A Reuel, N Haber, S Koyejo
Preprint · 2026
Stop Automating Peer Review Without Rigorous Evaluation
J Baumann, J Pei, S Koyejo, D Hovy
ICML 2026 Position Paper Track oral · 2026
The Leaderboard Illusion
S Singh, Y Nan, A Wang, D Dsouza, S Kapoor, A Üstün, S Koyejo, Y Deng, et al.
NeurIPS · 2025
Let's Measure Information Step-by-Step: AI-Based Evaluation Beyond Vibes
Z Robertson, S Koyejo
TMLR 2026 · 2025
Toward an Evaluation Science for Generative AI Systems
L Weidinger, ID Raji, H Wallach, M Mitchell, A Wang, O Salaudeen, et al.
arXiv preprint · 2025
Evaluation in the real world
Fairness, safety, and capability evaluated where AI systems actually operate, across domains, populations, and high-stakes settings.
9 papers
SWE-chat: Coding Agent Interactions From Real Users in the Wild
J Baumann, V Padmakumar, X Li, J Yang, D Yang, S Koyejo
arXiv preprint · 2026
In-Situ Behavioral Evaluation for LLM Fairness, Not Standardized-Test Scores
Z Tang, S Truong, D Owens, S Sharma, YJ Zhang, B Miranda, S Koyejo
arXiv preprint · 2026
Holistic evaluation of large language models for medical tasks with MedHELM
S Bedi, H Cui, M Fuentes, A Unell, M Wornow, JM Banda, N Kotecha, et al.
Nature Medicine · 2026
Why Do Safety Guardrails Degrade Across Languages?
M Zhang, A Patel, S Truong, S Koyejo
arXiv preprint · 2026
Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness
S Pfohl, N Harris, C Nagpal, D Madras, V Mhasawade, O Salaudeen, et al.
NeurIPS · 2025
AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration
A Zhou, K Wu, F Pinto, Z Chen, Y Zeng, Y Yang, S Yang, S Koyejo, J Zou, et al.
NeurIPS · 2025
The Sound of Syntax: Finetuning and Comprehensive Evaluation of Language Models for Speech Pathology
F Patel, DQ Nguyen, S Truong, J Vaynshtok, S Koyejo, N Haber
EMNLP · 2025
Development and evaluation of deep learning models for cardiotocography interpretation
N Chiou, N Young-Lin, C Kelly, J Cattiau, T Tiyasirichokchai, A Diack, et al.
npj Women's Health · 2025
Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models
S Truong, DQ Nguyen, TQ Nguyen, D Le, N Truong, T Quan, S Koyejo
NAACL 2024 Findings · 2024
Collaborate
Measurement science is a community effort.
Much of this work is joint with collaborators across labs and institutions. Join the conversation, follow new papers as they land, or reach out to work with us.