The AIMS Seminar.

Abstract

Benchmarks are necessary for healthcare evaluation, but are not sufficient for predicting deployment performance. Our position is that the evaluation–deployment gap arises not because of poorly designed benchmarks, but from implicit assumptions about how users interact with models that cannot be surfaced from benchmarks alone. To make this precise, we propose a classification of assumptions into two categories: task, which can be tested from conversation data alone, and outcome, which requires outcome data and behavioral studies for testing. Critically, outcome assumptions depend on human behavior, something that even well-designed benchmarks cannot directly observe. To demonstrate the operationality of this framework, we retrospectively analyze a healthcare RCT as a case study and find that the gap naturally separates into task and outcome gaps of roughly equal size. To address this, we make two contributions: first, we propose BenchmarkCards, an artifact that documents assumptions, and second, we propose staged evaluation, a procedure that systematically tests assumptions and evaluates performance.

Abstract

While aggregate leaderboard scores drive AI development, they contain substantial measurement noise whose sources and magnitudes remain unquantified, making it unclear when rankings reflect genuine capability differences versus evaluation artifacts. We introduce a framework for measuring the latent landscape in AI benchmark ecosystems. Applying Confirmatory Factor Analysis and Generalizability Theory to 4,000+ models from the Open LLM Leaderboard, we decompose sources of ranking variance, uncovering evidence of local dependence among leaderboard items, showing that contributor metadata explains more rank-relevant variance than architecture or deployment categories, and finding that the latent general-factor slope is far more stable than manifest-score scaling laws. We provide actionable diagnostics to determine how benchmark rankings can be trusted and how benchmark design can be improved.

Abstract

AI evals are ultimately inferences about a latent object (e.g., a model's capability), but oftentimes evals only report a proxy for the target of inference (e.g., accuracy on a benchmark). What justifies using accuracy as a proxy for a model's capability? Underlying this decision is a statistical model representing how data should relate to the target of inference, yet this crucial piece is often left implicit. In this talk, I will illustrate this idea using two examples. First, consider the task of assessing algorithmic monoculture (the extent to which an algorithmic ecosystem behaves similarly). A statement about excessive observed agreement is only meaningful relative to a null model of how much one would reasonably expect in the absence of true model dependence. Different, equally plausible, null models, can therefore produce sharply different conclusions about whether monoculture is present. Second, when assessing AI capability, benchmark performance becomes evidence of capability only through an assumed theory connecting question responses to latent abilities. Different theories of capability can lead to different interpretations of the same observed performance. Together, these cases suggest that AI evals are best understood as inference problems grounded in explicit, often normative, assumptions. I will discuss why this observation creates an obligation to articulate, justify, and test the models through which AI behavior is interpreted.

Abstract

The standard way to evaluate language models on subjective tasks is through pairwise comparisons: an annotator chooses the “better” of two model responses for a given prompt. These comparisons are then aggregated into a single ranking via the Bradley–Terry (BT) framework, forcing heterogeneous preferences into a total order and violating basic social-choice desiderata. In contrast, social choice theory provides an alternative approach called maximal lotteries, which aggregates pairwise preferences without imposing any assumptions on their structure. However, we show that maximal lotteries can be highly sensitive to heterogeneity among annotators and across prompts. We introduce robust lotteries, which optimize worst-case performance under plausible shifts in the preference data. On large-scale preference datasets, robust lotteries achieve more reliable win rate guarantees across the annotator distribution and recover a stable set of top performing models.