Research
Building the science of AI measurement.
AIMS advances the science of measuring and understanding AI systems, so that evaluations better inform research, deployment, and policy.
28 research entries
2026
Stop Automating Peer Review Without Rigorous Evaluation
A Measurement Science Roadmap: From Human Assessment to AI Evaluation
Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation
What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks
AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems
A Dataset for Modeling Iterative Problem-Solving
SWE-chat: Coding Agent Interactions From Real Users in the Wild
In-Situ Behavioral Evaluation for LLM Fairness, Not Standardized-Test Scores
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Holistic evaluation of large language models for medical tasks with MedHELM
Quantifying the Effect of Test Set Contamination on Generative Evaluations
Pretraining Scaling Laws for Generative Evaluations of Language Models
Let's Measure Information Step-by-Step: AI-Based Evaluation Beyond Vibes
Why Do Safety Guardrails Degrade Across Languages?
2025
Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness
Fantastic Bugs and Where to Find Them in AI Benchmarks
The Leaderboard Illusion
AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration
The Sound of Syntax: Finetuning and Comprehensive Evaluation of Language Models for Speech Pathology
Measurement to Meaning: A Validity-Centered Framework for AI Evaluation
Reliable and Efficient Amortized Model-based Evaluation
Development and evaluation of deep learning models for cardiotocography interpretation
Toward an Evaluation Science for Generative AI Systems
Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks
How Do Large Language Monkeys Get Their Power (Laws)?