Compass and certificate

A familiar exchange. A policy researcher says benchmarks are broken: the scores developers broadcast do not measure what matters once a system is deployed or regulated, and a number like MMLU certifies almost nothing about fitness for a real setting. A machine-learning researcher pushes back: benchmarks have driven a decade of real progress, the field would be flying blind without them, and the critique mistakes a working instrument for a useless one. Both are right. They also never move each other, because each is describing a different use of the same number, and sometimes a different number entirely.

The split is not rhetorical

The disagreement is not posturing. It tracks where people sit. In interviews with practitioners who had used benchmarks, or decided against using them, Hardy and colleagues (2025) found a consistent pattern: everyone uses benchmarks for relative comparison between models, but whether that signal counts as sufficient for a decision splits by setting. In academia, public benchmarks were treated as suitable for tracking research progress. In product and policy, the same benchmarks, including ones built in-house for a specific task, were often judged inadequate for substantive decisions, because their targets were neither well defined nor reflective of real-world use. The argument keeps its shape because the two camps stand in different rooms and ask the number for different things.

Why both are right

The deeper reason neither side gives ground is that the statistical case against benchmarks and the practical case for them are both correct. They do not conflict.

The case against is real: contamination, saturation, benchmark-targeted training, scores obtainable without doing the task, protocol bugs, and plain optimization pressure can all make a public number misleading.¹ On paper, a benchmark that is reused and optimized against should rot.

The practical case is real too, and the cleanest evidence is older. When Recht and colleagues (2019) built fresh test sets for ImageNet and CIFAR-10, absolute accuracy fell across the board, by eleven to fourteen percentage points on ImageNet. The level was fragile, just as the pessimist would predict. But the order of models held almost perfectly: the systems that ranked highest on the old test still ranked highest on the new one. Against that one stress, a fresh and harder draw from the same task, the level broke, and the direction did not.

That is the reconciliation, and also its limit. A benchmark can be a bad thermometer and a good compass: the absolute reading drifts while the ordering still points uphill. But the compass is not free. Recht et al. is a proof of possibility, not a theorem about benchmarks. Ordering survives some perturbations and breaks under others; benchmark-targeted training, reward hacking, and saturation can all swing the needle, and a magnetized compass sends a whole field up the wrong hill.² So the honest claim is narrow. Directional validity has to be earned, with different evidence than a certificate demands. The useful question about any benchmark is which property survives: the ordering, the level, or the construct.

The researcher defending benchmarks is defending the compass. The critic attacking them is attacking the certificate. Same device, different property.

The names we used to have

Evaluation theory once kept separate words for these uses. In program evaluation, Scriven (1967) distinguished formative evaluation, run during development to improve the thing, from summative evaluation, run at the end to judge whether it works. Formative use wants a direction and forgives a wobbly level. Summative use needs the level to mean something, which is the domain of construct validity, a century-old concern in measurement (Messick, 1989) already carried into machine learning (Jacobs and Wallach, 2021; Salaudeen et al., 2025; Freiesleben and Zezulka, 2025). The compass is formative evaluation in its everyday mode; the certificate is summative evaluation at its most demanding. Machine learning mostly proceeds without these terms, and the benchmark argument is part of the bill.³

Two jobs, one number

Part of the muddle is that "benchmark" names several things at once: a task distribution, a scoring rule, a public comparison, and sometimes a decision threshold. Compass and certificate are not properties of the dataset. They are claims about which decision the score is allowed to support.

Pull apart two things the conversation runs together. The first is who acts on the score: the team building the model, or someone outside deciding whether to trust it. The second is what the score is read as: a direction, or an absolute level. Put them on axes and formative and summative tend to fall on the diagonal. That is why the two get confused, not why they are defined that way; purpose sets the standard, and audience and reading are only its usual proxies. The real failure is migration. A number produced for one quadrant gets moved into another without earning the validity the new use requires.

A second confusion sits on top of the first, and it explains why the two camps so reliably miss. They are often not even discussing the same instrument. The critic attacks a real but narrow misuse, MMLU-as-certificate, and generalizes to "benchmarks are broken," a move with a canonical statement of its own (Raji et al., 2021). The researcher hears that and defends benchmark-as-compass, which the critic was never really attacking. Part of the talking-past is a confusion about use; part is a confusion about which object is in question. Both have to be named for the argument to dissolve.

Why this matters

The two uses do not only have different criteria. They have different victims, and that is what makes the policy complaint sharper than the product one.

A formative miss usually stays yours. You read the signal wrong, chase a gain that was not there, and lose a training cycle; the cost is internal and, at the formative stage, recoverable. Formative work runs fine on a rough signal, even a partly gamed one: being wrong is bounded, and it lands on the people who chose the risk. A summative miss usually belongs to someone else. A system that was not ready clears a bar, reaches users, and does whatever it was not ready to do, in a setting where the people bearing the consequence had no part in the measurement.⁴

Within the summative world, the stakes still vary, and governance is the unforgiving end. A product team that distrusts a benchmark has an escape: run a pilot, watch the telemetry, use the system on itself before committing. It can decide after trying. A regulator cannot use that escape in the same way. Its tools often act after the fact (monitoring, audits, recall, liability), or require public, standardized evidence up front rather than private trial-and-error. So it has to lean harder on evidence that is comparable, contestable, and robust to strategic optimization before the system is in use, or on mechanisms that keep deployment itself controlled and reversible. It also faces actors optimizing against whatever bar gets written into the rule. That is why the governance critique is the hardest version of the complaint: it needs certificate-grade evidence exactly where it is least able to improvise a workaround.

Back to the exchange

So neither camp is confused about benchmarks. They are talking past each other about what benchmarks are for. One side defends the compass function. The other attacks a certificate that, for open-ended systems in real deployment, we mostly do not yet know how to build.⁵ The move is to stop asking whether a benchmark is valid and start asking what it is valid for: which decision, at what cost of error, against what incentives. A compass only has to improve the search. A certificate has to justify trust. Treating those as one job is why the argument never ends. Building the second is the open problem: measurement that earns the word "certificate" by predicting behavior where the system is actually used.

Notes

1. In detail: scores inflate under contamination; cross-family comparison is confounded by differential training toward the test task, which is not cheating and still distorts the ranking (Dominguez-Olmedo, Dorner, and Hardt, 2025); close to half of widely used benchmarks no longer separate frontier models (Akhtar et al., 2026); agent scores can be reward-hacked without completing the task, and honest task bugs reshuffle rankings (Wang et al., 2026a, 2026b); and once a number gates a decision it gets optimized against, as Goodhart's law predicts (Manheim and Garrabrant, 2018).
2. Recht et al.'s robustness is to a fresh, harder test set drawn from the same construct; it does not extend to contamination, saturation, or training that targets the test task to different degrees across model families, which distorts the ranking directly (Dominguez-Olmedo, Dorner, and Hardt, 2025). So the open question is not whether benchmarks are compasses, but which benchmarks stay rank-stable under which perturbations. That is itself a measurement problem: it asks for the signal-to-noise properties of a benchmark as an instrument, not just its headline score.
3. The cut is being rediscovered under new names: a builder's view versus a decision-maker's view, the latter framed as a "decision-maker's dilemma" (Schwartz and Waters, 2026); first-party versus third-party evaluation (Reuel et al., 2026); public versus private benchmarks (Rajore et al., 2024); and calls to specify the deployment context before a metric is trusted. The common thread is that "eval" is one word for many constructs, a textbook jingle fallacy.
4. The asymmetry is about where the cost lands at the moment the mistake is made, not a claim that formative errors never reach users. Benchmark-chasing can shape a product that later harms people, but that harm passes through a summative ship decision; the formative error becomes consequential only when a summative gate fails to catch it. Some summative decisions are low-stakes.
5. Certificates do exist in narrow, regulated regimes (medical devices, aviation, some security and audit settings), and partial instruments like model cards and third-party audits are emerging. What is missing is a general, reusable certificate for open-ended capability or safety in a deployment context: as standardized and comparable as a benchmark, but with the validity that benchmarks often lack.

AI use disclosure: The authors used Claude (Anthropic) to copyedit author-written text: grammar, spelling, and sentence-level phrasing. No analysis or citations were model-generated. The authors reviewed all suggestions and take full responsibility for the content.

References

• Akhtar, M., Reuel, A., Soni, P., Ahuja, S., Ammanamanchi, P. S., Rawal, R., Zouhar, V., Yadav, S., Whitehouse, C., Ki, D., Mickel, J., Choshen, L., Šuppa, M., Batzner, J., Chim, J., Sania, J., Long, Y., Rahmani, H. A., Knight, C., Nan, Y., Raj, J., Fan, Y., Singh, S., Sahoo, S., Habba, E., Gohar, U., Pawar, S., Scholz, R., Subramonian, A., Ni, J., Kochenderfer, M., Koyejo, S., Sachan, M., Biderman, S., Talat, Z., Ghosh, A., and Solaiman, I. (2026). When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation. ICML 2026. arXiv:2602.16763.
• Dominguez-Olmedo, R., Dorner, F. E., and Hardt, M. (2025). Training on the Test Task Confounds Evaluation and Emergence. ICLR 2025. arXiv:2407.07890.
• Freiesleben, T., and Zezulka, S. (2025). The Benchmarking Epistemology: Construct Validity for Evaluating Machine Learning Models. arXiv:2510.23191.
• Hardy, A., Reuel, A., Jafari Meimandi, K., Soder, L., Griffith, A., Asmar, D. M., Koyejo, S., Bernstein, M. S., and Kochenderfer, M. J. (2025). More than Marketing? On the Information Value of AI Benchmarks for Practitioners. IUI 2025, 1032–1047. ACM. doi:10.1145/3708359.3712152. arXiv:2412.05520.
• Jacobs, A. Z., and Wallach, H. (2021). Measurement and Fairness. FAccT 2021. arXiv:1912.05511.
• Manheim, D., and Garrabrant, S. (2018). Categorizing Variants of Goodhart's Law. arXiv:1803.04585.
• Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 13–103). American Council on Education / Macmillan.
• Raji, I. D., Bender, E. M., Paullada, A., Denton, E., and Hanna, A. (2021). AI and the Everything in the Whole Wide World Benchmark. NeurIPS Datasets and Benchmarks 2021. arXiv:2111.15366.
• Rajore, T., Chandran, N., Sitaram, S., Gupta, D., Sharma, R., Mittal, K., and Swaminathan, M. (2024). TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs. arXiv:2403.00393.
• Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. (2019). Do ImageNet Classifiers Generalize to ImageNet? ICML 2019. arXiv:1902.10811.
• Reuel, A., Ghosh, A., Chim, J., Tran, A., Long, Y., Mickel, J., Gohar, U., Yadav, S., Ammanamanchi, P. S., Allaham, M., Rahmani, H. A., Akhtar, M., Friedrich, F., Scholz, R., Riegler, M. A., Batzner, J., Habba, E., Saxena, A., Kornilova, A., Wei, K., Soni, P., Mathew, Y., Klyman, K., Sania, J., Sahoo, S., Beyer Bruvik, O., Sadeghi, P., Goswami, S., Wang, A., Jernite, Y., Talat, Z., Biderman, S., Kochenderfer, M., Koyejo, S., and Solaiman, I. (2026). Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations. ICML 2026. arXiv:2511.05613.
• Salaudeen, O., Reuel, A., Ahmed, A., Bedi, S., Robertson, Z., Sundar, S., Domingue, B., Wang, A., and Koyejo, S. (2025). Measurement to Meaning: A Validity-Centered Framework for AI Evaluation. arXiv:2505.10573.
• Schwartz, R., and Waters, G. (2026). Real-World AI Evaluation: How FRAME Generates Systematic Evidence to Resolve the Decision-Maker's Dilemma. arXiv:2603.13294.
• Scriven, M. (1967). The Methodology of Evaluation. In R. E. Stake (Ed.), Perspectives of Curriculum Evaluation (AERA Monograph Series on Curriculum Evaluation, No. 1). Chicago: Rand McNally.
• Wang, H., Li, H., Mang, Q., Cheung, A., Sen, K., and Song, D. (2026a). Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack. arXiv:2605.12673.
• Wang, J., Bianchi, F., Zhu, S., Nie, F., Kwon, Y., Dhingra, B., and Zou, J. (2026b). Automated Benchmark Auditing for AI Agents and Large Language Models. arXiv:2605.26079.