11 Design
By the end of this chapter, the reader will be able to:
- Explain why Goodhart’s Law is a formal game-theoretic phenomenon rooted in the gap between a construct \(\mathcal{C}\) and any single distribution \(D\) that operationalizes it, and classify its four variants.
- Distinguish the three objects an evaluator might keep private — the construct \(\mathcal{C}\), the operationalizing distribution \(D\), and the realized set \(S\) — and classify mechanisms by which object they hide.
- Formalize AI evaluation as a Stackelberg game and prove that a transparent benchmark fails, while randomization (hiding the realized draw from \(D\)) aligns incentives up to the builder’s belief error \(\operatorname{TV}(\mu, \tilde{\pi}_E)\).
- Explain how reusable holdouts and the Ladder algorithm hide the set \(S\) and bound information leakage, and why dynamic benchmarks that refresh \(S\) without hiding \(D\) remain exploitable.
- Analyze the information-variance tradeoff that hiding \(D\) creates in repeated evaluation, and derive distribution correction (and the optimal size \(k^*\)) as the response.
- State the construct-validity protocol — publish \(\mathcal{C}\) and public proxies while keeping \(D, S\) private — and the three properties of a durable evaluator (validity, durability, faithful execution).
- Model metric design as a principal-agent problem and apply positional representation and proportionality criteria to select representative benchmark subsets.
This chapter can be covered in 3 lectures (75-90 minutes each):
Lecture 1: From Goodhart to “What Are We Hiding?”
- Goodhart’s Law and its four variants (15 min)
- Construct, distribution, set: the object-first framing (20 min)
- The Stackelberg evaluation game; why a static benchmark fails (25 min)
- Hands-on: deterministic vs. randomized evaluation simulation (15 min)
Lecture 2: Hiding the Set and Hiding the Distribution
- Reusable holdouts, the Ladder algorithm, and dynamic benchmarks (20 min)
- One-shot alignment via randomization; the information-variance tradeoff (25 min)
- Distribution correction and the optimal evaluation size (20 min)
- Hands-on: Pareto frontier and correction simulations (10 min)
Lecture 3: Construct, Metrics, and Synthesis
- The construct-validity protocol: publishing \(\mathcal{C}\) and proxies (15 min)
- Metric design as a principal-agent problem; reveal, conceal, garble (25 min)
- Representative benchmark selection via social choice theory (20 min)
- Three properties of a durable evaluator; design principles (15 min)
This chapter introduces game-theoretic notation: \(F\) (task universe), \(F_E/F_M\) (evaluator/builder task sets), \(\pi_E/\pi_M\) (sampling distributions), \(f(\theta)\) (task performance), \(u_E\) (evaluator utility), \(\mu\) (social-relevance distribution), \(\Delta_t\) (misalignment), \(\rho\) (correction rate), and \(k\) (tasks sampled per round). It also introduces the object-first vocabulary of privacy in evaluation: the construct \(\mathcal{C}\), an operationalizing distribution \(D\) (formally the evaluator’s sampling distribution \(\pi_E\)), a realized evaluation set \(S\), the construct’s full operationalization \(\mu\) (the social-relevance weighting; \(\text{Uniform}(F)\) as a special case), and public proxy distributions \(D', D''\). See the notation appendix for the complete notation reference.
11.1 When Measurement Becomes a Target
“When a measure becomes a target, it ceases to be a good measure.” — Charles Goodhart (1975)
In previous chapters, we treated evaluation as a statistical problem: how to estimate latent abilities from noisy observations (Chapter 3), how to quantify measurement precision (Chapter 5), and how to assess whether a benchmark measures what it claims to measure (Chapter 1, Chapter 9). Throughout, we implicitly assumed that the data-generating process is fixed — that the act of measurement does not change the phenomenon being measured.
This assumption breaks down once benchmarks become influential. When a developer’s reputation, funding, or regulatory standing depends on benchmark scores, the developer has every incentive to optimize specifically for the benchmark — not for the broader capability it claims to measure. The data-generating process is no longer fixed: it shifts in response to the measurement itself. This is Goodhart’s Law, and in AI evaluation it is not a metaphor but a demonstrable, recurring phenomenon.
Concrete examples. The Foundation Model Transparency Index (FMTI) saw scores jump dramatically after its first release, not because companies became more transparent, but because they learned which specific indicators were measured and optimized their disclosures accordingly. Chatbot Arena has faced concerns about strategic submission: developers can selectively deploy models optimized for the types of queries that appear on the platform. And benchmark contamination — where training data includes evaluation items — is rational behavior for a developer whose market value depends on leaderboard position (Section 1.3.3).
Manheim and Garrabrant (2018) identify four variants of the Goodhart effect, each with a distinct causal mechanism:
Regressional Goodhart: The proxy \(\hat{u}\) and the true objective \(u\) are correlated but not identical. Optimizing \(\hat{u}\) overshoots because extreme values of \(\hat{u}\) tend to arise from noise, not from genuinely extreme \(u\). This is the measurement error story from Chapter 5: a benchmark with \(\alpha = 0.7\) captures 70% true variance and 30% noise, so selecting the top-scoring model partially selects for lucky noise.
Extremal Goodhart: The relationship between \(\hat{u}\) and \(u\) that holds in the bulk of the distribution breaks down in the tails. A benchmark that reliably ranks typical models may fail catastrophically when applied to a model specifically engineered to maximize the benchmark score.
Causal Goodhart: The proxy and the objective share a common cause. Optimizing the proxy can break this causal link. If “passes safety benchmark” and “is actually safe” are both caused by “was trained carefully,” a developer can find shortcuts to the benchmark that bypass the common cause.
Adversarial Goodhart: An agent actively exploits the gap between proxy and objective. This is the regime that dominates in competitive AI evaluation, and the focus of this chapter.
The shift from the earlier measurement chapters to this one is a shift in the source of the threat. Measurement error (Chapter 5) is stochastic. Validity threats (Section 1.3) are systematic but unintentional. Strategic manipulation is deliberate and adaptive: the DGP itself changes in response to the measurement. This connects to the literature on performative prediction (Perdomo et al. 2020), where the act of deploying a model changes the distribution it operates on, and to strategic classification (Hardt et al. 2016), where agents manipulate their features to achieve favorable outcomes.
Underlying all four variants is a single structural fact: the construct we care about — say, “mathematical capability” — is always broader than any distribution of tasks we can actually pose. We instantiate “math ability” with one particular distribution over problems, and a rational builder optimizes for that distribution, not for the construct behind it. The rest of this chapter takes this gap as its organizing question. Rather than first asking which mechanism an evaluator should use, we ask what object the evaluator must keep private to preserve the meaning of the score (Section 11.2). That question orders the chapter: we escalate from hiding the realized test set (Section 11.4), to hiding the distribution that generates it (Section 11.5), to publishing the construct itself alongside public proxies (Section 11.6).
11.2 What Are We Trying to Hide?
Before choosing a mechanism, an evaluator should answer a prior question: what object are we trying to keep private? Different protocols protect different things, and once we name the object, the choice between deterministic and randomized machinery often settles itself. We distinguish three objects, nested from concrete to abstract.
- A construct \(\mathcal{C}\) is the capability we intend to measure — e.g., “mathematical reasoning.” It is a latent attribute (Chapter 1), not a finite object.
- A distribution \(D\) is an operationalization of \(\mathcal{C}\): a way of generating tasks that we accept as measuring it. FrontierMath is a sample from one such distribution; the process that could generate another FrontierMath-like exam is \(D\) itself.
- A set \(S \sim D\) is a realized, finite sample of tasks — the specific items on today’s exam.
A single construct admits many distributions. GSM8K and FrontierMath both claim to measure mathematical capability, yet they induce very different task distributions \(D'\) and \(D\); both are operationalizations of the same \(\mathcal{C}\). This multiplicity is not a defect — it is exactly what lets us reason about validity and, eventually, what lets us publish a public proxy while keeping a private benchmark (Section 11.6).
The source of the incentive problem. The construct is always broader than any distribution that operationalizes it: \(\mathcal{C}\) is an idealization, \(D\) a finite and fallible instrument. We say we measure “math ability” (\(\mathcal{C}\)) but we contract on performance under one distribution (\(D\)). A strategic builder optimizes for \(D\), not \(\mathcal{C}\). This \(\mathcal{C} \supsetneq D\) gap is the engine behind every Goodhart variant of Section 11.1.
Connecting to the formal model. The game-theoretic notation of this chapter is a formalization of these three objects. The social-relevance distribution \(\mu\) that weights the evaluator’s utility \(u_E(\theta) = \mathbb{E}_{f \sim \mu}[f(\theta)]\) — how much each task actually matters in deployment — is the construct \(\mathcal{C}\) fully operationalized, “what we actually care about” (the uniform weighting \(\mu = \text{Uniform}(F)\) is the simplest special case). The sampling distribution \(\pi_E\) is the operationalizing distribution \(D\), and a realized draw \(S = M(F_E, \omega)\) is the set \(S\). In this language the incentive problem is simply the statement \(\pi_E \neq \mu\): the distribution we can sample is a biased, narrow shadow of the construct we care about.
With the objects named, mechanisms sort cleanly by what they hide.
| Mechanism | Object hidden | What leaks | Durable? | Deterministic OK? |
|---|---|---|---|---|
| Static public benchmark | nothing | \(S\), and hence \(D\) | no | — |
| Reusable holdout / Ladder (Section 11.4.2) | the set \(S\) | score signal, slowly | for a while | yes |
| Dynamic benchmark (Section 11.4.3) | today’s \(S\) | \(D\), over rounds | no | yes |
| Randomized evaluation (Section 11.5) | the distribution \(D\) | \(D\), slowly | yes, with correction | no |
| Construct-validity protocol (Section 11.6) | \(D\) and \(S\) | only proxy \(D'\) | yes | either |
Two columns deserve comment. Deterministic OK? records the chapter’s central reconciliation: deterministic machinery is fine when the object you protect is the set (a fixed holdout can be kept secret and reused), but it fails when the object is the distribution — Section 11.5 shows that hiding \(D\) requires randomization. The right mechanism depends on the object, not on a blanket preference for randomness. Durable? records whether the protection survives a builder who plays the game many times.
An orthogonal axis: how much leaks. Independent of which object you hide is how much you reveal about it per round. The Ladder algorithm (Section 11.4.2) is best understood as a control on this second axis: it can wrap either a hidden set or a hidden distribution, releasing a sliver of signal only when a submission genuinely improves. “What to hide” and “how much to leak” are separate design decisions, and conflating them is what makes the deterministic-versus-randomized debate feel irreconcilable.
Three properties. Naming the objects lets us state what a good evaluator should guarantee. We return to these in Section 11.9, but they motivate the whole chapter:
- Validity. The reported score carries the intended construct interpretation — it tracks \(\mathcal{C}\), not merely \(D\) (Chapter 1).
- Durability. The benchmark stays informative over time, because \(D\) (and ideally \(S\)) remains private and therefore hard to game.
- Faithful execution. The evaluator actually runs the promised procedure — the same sampling, scoring, and reporting for everyone, with no special treatment. This merges what one might separately call neutrality and integrity: both reduce to honestly executing the announced protocol.
Roadmap. The chapter now escalates through the objects. Section 11.4 hides the set \(S\) — reusable holdouts, the Ladder algorithm, and dynamic benchmarks — and shows where each stops working. Section 11.5 hides the distribution \(D\) through randomization, develops the information-variance tradeoff that hiding \(D\) creates, and resolves it with distribution correction. Section 11.6 then turns the \(\mathcal{C} \supsetneq D\) gap from a liability into a tool: by publishing the construct and public proxies while keeping \(D\) and \(S\) private, an evaluator can certify capability without ever exposing the benchmark.
11.3 The Evaluation Game
We now formalize the strategic interaction between an evaluation designer and a model builder as a Stackelberg game. The designer moves first by committing to an evaluation mechanism; the builder responds by training a model. The key question: under what conditions can a benchmark mechanism incentivize builders to improve performance on the full task distribution, rather than merely optimizing for the specific tasks being measured?
11.3.1 Setup: Evaluator and Builder
Let \(\Theta\) denote the space of all possible models. For a given model \(\theta \in \Theta\), let \(f(\theta) \in [0, 1]\) denote the model’s performance on task \(f\). The universe of all possible tasks is the finite set \(F\) with \(|F| = N\).
The evaluation designer’s utility for a model \(\theta\) is its performance averaged under the social-relevance distribution \(\mu\) over tasks: \[ u_E(\theta) = \sum_{f \in F} f(\theta)\,\mu(f) = \mathbb{E}_{f \sim \mu}[f(\theta)] \] where \(\mu(f)\) is how socially relevant task \(f\) is — e.g., the probability that \(f\) is encountered in real deployment. The designer acts as a social planner seeking models that perform broadly across what matters, rather than narrowly. The uniform weighting \(\mu = \text{Uniform}(F)\), under which \(u_E(\theta) \propto \sum_{f} f(\theta)\), is the simplest special case.
Each party acquires tasks by sampling from \(F\): the designer draws tasks according to \(\pi_E\) over \(F\), forming the evaluation set \(F_E\), while the builder draws according to \(\pi_M\), forming the training set \(F_M\). In practice, \(\pi_M\) is approximately uniform (builders have broad access to training data), while \(\pi_E\) is typically biased toward tasks that are easy to construct, grade, or that reflect particular evaluation priorities — in general \(\pi_E \neq \mu\).
The game proceeds in three stages, parameterized by a mechanism \((M, r)\) where \(M\) is a sampling function and \(r: \Theta \times \mathcal{P}(F) \to \mathbb{R}\) is a reward function:
Designer’s move (ex-ante): The evaluation designer publishes the mechanism \((M, r)\). If \(M\) is randomized, the designer privately draws randomness \(\omega\) and computes \(S^* = M(F_E, \omega)\), withholding the realization. If \(M\) is deterministic, \(S^* \subseteq F_E\) is fixed.
Builder’s move: The builder observes the mechanism \((M, r)\) but not the realized set \(S^*\), and selects a model \(\theta^* \in \Theta\).
Evaluation stage (ex-post): The designer publishes the score \(r(\theta^*, S^*)\). The builder’s payoff is \(r(\theta^*, S^*)\); the designer’s payoff is \(u_E(\theta^*)\).
The misalignment between the builder’s incentive (performance on \(S^*\)) and the designer’s objective (performance on all of \(F\)) creates the Goodhart problem.
11.4 Hiding the Set
The mildest thing an evaluator can keep private is the realized test set \(S\). The construct and even the sampling distribution may be public; only the specific items on today’s exam are secret. This is the regime of held-out test sets, and — crucially — it is a regime where deterministic machinery works: a fixed, secret holdout, reused carefully across many submissions, can preserve a leaderboard for a long time. The catch is that every score computed on \(S\) leaks a little information about it, so the central design problem is bounding that leak. We first see why the naive version (publishing \(S\)) fails, then how the Ladder algorithm hides \(S\) while bounding leakage, and finally why refreshing \(S\) — dynamic benchmarking — does not reach the harder goal of hiding \(D\).
11.4.1 The Static Benchmark and Why It Fails
Consider first a transparent mechanism: the builder observes the realized evaluation set \(S^* \subseteq F_E\) before choosing a model. A published, fixed benchmark is the canonical example.
If the builder observes the realized set \(S^*\) and the reward is non-degenerate on it (e.g., additive \(r(\theta, S) = \sum_{f \in S} f(\theta)\)), the builder’s best response is: \[ \theta^* \in \arg\max_{\theta \in \Theta} \sum_{f \in S^*} f(\theta) \] Whenever \(|S^*| < |F|\), there exist a task universe \(F\) and social-relevance weighting \(\mu\) for which this is socially suboptimal: \(u_E(\theta^*) < \max_\theta u_E(\theta)\). The mechanism provides no incentive for performance on tasks outside \(S^*\).
Proof. The builder observes \(S^*\) and maximizes the known objective, so tasks in \(F \setminus S^*\) do not affect the score. Place a socially decisive task \(f' \notin S^*\) (some \(\mu(f') > 0\) with \(f'\) maximized away from \(\theta^*\)); then \(\theta^*\) misses it. \(\square\)
This is exactly Goodhart’s adversarial variant: the builder exploits the observed structure of the evaluation to specialize, the construct underrepresentation problem of Section 1.3.2 seen through a strategic lens.
It is worth being precise about what fails here. The culprit is transparency, not determinism. A benchmark whose sampling rule is deterministic but whose realized set \(S^*\) is kept hidden does not fall under the proposition: the builder is then Bayesian over \(S^*\), and the outcome is governed by the single-sample alignment bound of Section 11.5.1 via the builder’s belief \(\tilde{\pi}_E\). Randomization is one way to enforce the necessary secrecy; a hidden but deterministic holdout is another — with the crucial caveat that any leak collapses it back to the transparent case (the failure mode the Ladder of Section 11.4.2 is designed to slow). We visualize the transparent-versus-randomized contrast in Section 11.5.1.
11.4.2 The Ladder Algorithm and Reusable Holdouts
If a static public set fails because \(S\) is exposed, the obvious fix is to keep \(S\) secret and let builders query it only through scores. But scores themselves leak: a builder who submits many models against a fixed holdout can overfit it through the published numbers alone, even without ever seeing an item — the adaptive data analysis problem. Blum and Hardt (2015)’s Ladder algorithm bounds this leak. It releases a new score only when a submission improves on the best score so far by more than a noisy margin; otherwise it re-reports the previous best. Because most submissions are not genuine improvements, almost no information about \(S\) flows back to the builder.
Maintain a running best published score \(b_0 = -\infty\). For each submission \(\theta_t\) (\(t = 1, 2, \ldots\)):
- Compute the holdout score \(r_H(\theta_t) = \frac{1}{|S|}\sum_{f \in S} f(\theta_t)\) on the secret set \(S\).
- Draw noise \(\xi_t \sim \text{Laplace}(0, 1/\varepsilon)\) and a threshold \(\eta > 0\).
- If \(r_H(\theta_t) > b_{t-1} + \eta + \xi_t\): publish \(b_t \leftarrow \lfloor r_H(\theta_t) \rceil_\eta\) (round to step \(\eta\)). Else: publish \(b_t \leftarrow b_{t-1}\) (no new information).
We can extend this into a mechanism that gates information on the builder’s behavior rather than only on improvement, by comparing the secret holdout against a public reference set.
The evaluator publishes a reference set \(S_0 \subset F\) of size \(n_0\), sampled from \(\pi_E\). At each round \(t\):
- Draw a holdout set \(S_t\) of size \(k\) from \(\pi_E\) and noise \(\xi_t \sim \text{Laplace}(0, 1/\varepsilon)\).
- Compute the reference score \(r_T(\theta_t) = \frac{1}{n_0}\sum_{f \in S_0} f(\theta_t)\) and the holdout score \(r_H(\theta_t) = \frac{1}{k}\sum_{f \in S_t} f(\theta_t)\).
- Threshold test: If \(|r_H(\theta_t) + \xi_t - r_T(\theta_t)| < \eta\), publish \(r_T(\theta_t)\) (“silent” round). Otherwise, publish \(r_H(\theta_t) + \xi_t\) (“alarm” round).
The mechanism is self-correcting: it reveals less information precisely when the builder is behaving well (\(\Delta_{\text{game}} \approx 0\), so the alarm fires only due to noise, with probability \(p_{\text{alarm}} = e^{-\varepsilon\eta}\)), and reveals more when the builder is gaming (the score discrepancy reliably triggers the alarm). The effective leakage per round for a non-gaming builder is: \[ \lambda_{\text{out}}^{\text{holdout}} \approx e^{-\varepsilon\eta} \cdot k \] Setting \(\varepsilon\eta = 3\) yields a \(\sim 20\times\) slowdown in the builder’s learning rate.
This is a differential-privacy guarantee in disguise (Dwork et al. 2015; Dwork and Roth 2014): the Laplace threshold is the Sparse Vector mechanism, and it makes the published transcript insensitive to any single held-out item, bounding the max-information between the builder’s observations and \(S\). Hiding the set is, in this sense, a solved problem: a secret \(S\) can be reused for a bounded number of informative queries.
A perfectly hidden, leak-bounded set protects only \(S\). If the distribution \(D = \pi_E\) is biased and fixed, a builder who never sees a single item can still specialize to it — because \(D\) is public, or quickly inferred. The builder’s optimal model given their belief \(\hat{\pi}_{E}\) about the distribution is \(\theta^* = \arg\max_\theta \mathbb{E}_{f \sim \hat{\pi}_{E}}[f(\theta)]\), regardless of how little leaks about any particular \(S\). Hiding the set caps overfitting to particular items; it does nothing about overfitting to the distribution. That requires hiding \(D\) itself — the subject of Section 11.5.
11.4.3 Dynamic Benchmarks
A popular alternative to a single hidden holdout is to refresh it: draw a fresh set \(S' \sim D\) each period and evaluate on it immediately. DynaBench (Kiela et al. 2021), LiveBench (White et al. 2025), and the rolling battles of Chatbot Arena (Chiang et al. 2024) are all, in this framing, repeated sampling of \(S' \sim D\) with immediate disclosure. Refreshing defeats item-level contamination: yesterday’s leaked items are worthless because tomorrow’s set is new.
But dynamic refresh hides the set, not the distribution, and two limitations follow.
First, it does not hide \(D\). Each disclosed \(S'\) is a fresh sample from the same \(D\); across rounds the builder accumulates samples and can estimate \(D\) to arbitrary precision (the posterior-concentration argument of Section 11.5.3.2). A reader may ask: why repeatedly draw a new \(S'\) at all — why not build two sets \(S\) and \(S'\) once, publish \(S'\), and permanently hold out \(S\)? That design is exactly the reusable holdout of Section 11.4.2, and the held-out \(S\) is consumed at the Ladder-bounded leak rate; refreshing merely trades a slowly-leaking fixed set for a stream of fully-disclosed ones. Either way \(D\) becomes public or quickly learned. Durability against distributional gaming requires hiding \(D\), not cycling \(S\).
Second, if \(D\) deviates from the construct, refreshing cannot help. Suppose \(D\) over-weights, say, competition-style problems relative to the construct weighting \(\mu\). Every \(S' \sim D\) inherits that bias. A builder who tracks \(D\) — easy, since it leaks — specializes to it, and the resulting misalignment does not vanish with the refresh rate: it is governed by how far \(D\) sits from \(\mu\), on the order of \(\operatorname{TV}(\mu, D)\), no matter how often \(S\) is renewed (Exercise 9). Dynamic benchmarks fix contamination; they do not fix construct misalignment.
These two failures — a learnable \(D\) and a biased \(D\) — are exactly what the next sections address: Section 11.5 hides \(D\) and corrects it toward \(\mu\), and Section 11.6 confronts the \(D \neq \mathcal{C}\) gap directly.
11.5 Hiding the Distribution
Hiding the set caps how much a builder can overfit particular items, but it leaves the distribution \(D = \pi_E\) exposed — and a public distribution is a public target. To incentivize broad capability we must keep \(D\) itself uncertain to the builder. This section shows that hiding \(D\) (i) requires randomization — no deterministic mechanism suffices — (ii) creates an unavoidable tradeoff between score variance and information leakage, and (iii) is made durable by correcting \(D\) toward the construct \(\mu\) faster than the builder can learn it.
11.5.1 One-Shot Alignment via Randomization
While transparent mechanisms fail, randomization offers a path forward. When the builder faces genuine uncertainty about which task will be scored, the strategic landscape changes fundamentally. The simplest such mechanism scores a single random task.
\(S(F_E, \omega)\) draws one task \(s\) uniformly from \(F_E\), and the reward is \(r(\theta, \{s\}) = s(\theta)\). The builder, not knowing which task will be drawn, is Bayesian: it holds a belief \(\tilde{\pi}_E\) over which tasks the evaluator might score, where \(\tilde{\pi}_E(f)\) is the builder’s subjective probability that \(f\) is the sampled task.
Because the builder cannot identify the realized task, it can only optimize against its belief \(\tilde{\pi}_E\).
Under the single-sample mechanism, the Bayesian builder’s best response is \[ \theta^M \in \arg\max_{\theta \in \Theta} \mathbb{E}_{f \sim \tilde{\pi}_E}[f(\theta)]. \] The builder optimizes broadly, over its whole belief about what might be tested, rather than for any fixed set.
The value of this is captured by a single bound: the evaluator’s loss is controlled by how far the builder’s belief sits from social relevance.
Under the single-sample mechanism, the evaluator’s suboptimality is bounded by the total variation distance between the builder’s belief and the social-relevance distribution: \[ \max_{\theta \in \Theta} u_E(\theta) - u_E(\theta^M) \;\leq\; 4B \cdot \operatorname{TV}(\mu, \tilde{\pi}_E), \] where \(B = \sup_{f \in F;\, a,b \in \Theta} |f(a) - f(b)|\) bounds the spread of task scores.
Proof sketch. The builder maximizes \(\mathbb{E}_{\tilde{\pi}_E}[f(\theta)]\) while the evaluator cares about \(\mathbb{E}_{\mu}[f(\theta)] = u_E(\theta)\). Adding and subtracting the belief-weighted objective at the builder’s and the optimal model and using optimality of \(\theta^M\) for the former bounds the gap by \(2\sup_\theta |\mathbb{E}_\mu[f(\theta)] - \mathbb{E}_{\tilde{\pi}_E}[f(\theta)]|\); since scores lie in an interval of width \(\le B\), this discrepancy is at most \(2B\operatorname{TV}(\mu, \tilde{\pi}_E)\). \(\square\)
This bound is the chapter’s central one-shot result, and it makes the privacy intuition exact: misalignment is bounded by the distance between what the builder believes will be tested and what society actually cares about. Two corollaries pin down the endpoints.
If the builder’s belief matches social relevance, \(\tilde{\pi}_E = \mu\), then \(\operatorname{TV} = 0\) and \(\theta^M\) maximizes \(u_E\) exactly. A well-informed builder who believes the evaluator is equally well-informed is perfectly aligned. (The uniform special case \(\mu = \text{Uniform}(F)\) recovers the familiar statement that the builder’s expected reward is proportional to \(u_E\).)
If \(F_E \subset F_M\) and \(\tilde{\pi}_E(f) > 0\) for some \(f \notin F_E\), the single-sample mechanism incentivizes the builder over a strictly larger set of tasks than any transparent mechanism — including tasks the evaluator cannot even sample, as long as the builder believes they might be tested.
The practical upshot: under randomization the builder’s best move is to optimize for its honest belief about what matters, which — for a well-resourced builder whose belief is broad — means training broadly rather than gaming a known set. A boundedly rational builder that can only estimate \(\mathbb{E}_{\tilde{\pi}_E}[f(\theta)]\) from a finite sample \(F_M \sim \pi_M\) does importance-weighted empirical risk minimization; by uniform convergence (Vapnik 1998) its worst-case guarantee is monotone in sample size, so it has no statistical reason to discard tasks — no incentive to “game” by narrowing.
11.5.2 Discussion: What Randomization Buys
The one-shot alignment result reveals a fundamental tension with the statistical efficiency goals of earlier chapters. In Section 6.2.1 and Section 6.2.2, we showed that targeted evaluation — choosing items to maximize Fisher information about \(\theta\) — is statistically optimal. But targeted evaluation requires revealing information about which tasks the evaluator considers informative, which creates exploitable structure.
Randomized evaluation is not optimal for estimation precision but is optimal for incentive alignment. This tension between statistical efficiency and strategic robustness is a recurring theme:
- Computerized adaptive testing (Section 6.2.2) reveals the evaluator’s information about ability through the item selection strategy, making the evaluator’s priorities transparent.
- Bayesian persuasion (Kamenica and Gentzkow 2011) provides the information design framework: the evaluator is a sender choosing an information structure, and the builder is a receiver who updates and best-responds. The evaluator’s problem is to design a signal that induces the builder to take the action the evaluator prefers.
The resolution, as we develop in the next two sections, is that the evaluator must invest in correction to make the inevitable information leakage harmless.
Figure. Under a transparent mechanism (left), the builder observes the evaluation set and concentrates effort on the \(k=10\) known tasks, neglecting the rest — high benchmark score, low true utility. Under a randomized mechanism (center), facing uncertainty about what will be scored, the builder spreads effort broadly. The right panel compares true utility \(u_E = \mathbb{E}_{f \sim \mu}[\text{performance}]\) under the social-relevance weighting \(\mu\): randomization yields substantially higher utility despite lower peak performance on any single task.
11.5.3 The Information-Variance Tradeoff
The single-sample alignment bound (Section 11.5.1) showed that randomization aligns incentives in a one-shot game, up to the builder’s belief error \(\operatorname{TV}(\mu, \tilde{\pi}_E)\). But benchmarks operate as repeated games: weekly leaderboards, monthly submissions, quarterly benchmark releases. In repeated evaluation, randomization alone fails because information leaks — the builder’s belief \(\tilde{\pi}_E\) drifts toward the true \(\pi_E\), and if \(\pi_E \neq \mu\) the bound degrades.
11.5.3.1 Repeated Evaluation and Information Leakage
At each round \(t = 1, 2, \ldots\):
- The evaluator draws a fresh evaluation set \(F_E^{(t)}\) by sampling from \(F\) according to \(\pi_E\).
- The evaluator samples \(k\) tasks from \(F_E^{(t)}\) via mechanism \(M\) and publishes the score.
- The builder observes the \(k\) evaluated tasks, updates their estimate \(\hat{\pi}_{E,t}\) of the evaluation distribution, and selects model \(\theta_t\).
For a size-\(k\) subset drawn without replacement from \(F_E\) with per-task score variance \(\sigma^2\), the average reward has the finite-population variance \[ \operatorname{Var}(\hat{r}_k) = \frac{\sigma^2}{k}\cdot\frac{|F_E| - k}{|F_E| - 1} \approx \frac{\sigma^2}{k} \quad (k \ll |F_E|). \] To reliably distinguish two models with performance gap \(\Delta\) at 95% confidence requires \(k \geq 4\sigma^2/\Delta^2\). As models improve and converge (\(\Delta \to 0\)), the required sample size grows quadratically. This creates inexorable market pressure to increase \(k\).
But increasing \(k\) to reduce variance simultaneously increases the rate at which the builder accumulates information about \(\pi_E\). The builder’s information set at time \(t\) is \(\mathcal{I}_t = \bigcup_{i=1}^{t-1} S_i\), where \(S_i\) is the set of tasks sampled in round \(i\). The leakage is \(L_t = |\mathcal{I}_t|/|F|\).
11.5.3.2 Posterior Concentration and Incentive Misalignment
As the builder observes sampled tasks across rounds, their estimate \(\hat{\pi}_{E,t}\) of the evaluation distribution concentrates around the true \(\pi_E\). By standard results in Bayesian nonparametrics, the posterior concentration rate is \(O(d_{\text{eff}} \log m / m)\) where \(m\) is the number of observations and \(d_{\text{eff}}\) is the effective dimension of the distribution class.
Let \(\hat{\pi}_{E,t}\) denote the builder’s posterior mean estimate of \(\pi_E\) at time \(t\). The builder’s optimal strategy is \(\theta^*_t = \arg\max_\theta \mathbb{E}_{f \sim \hat{\pi}_{E,t}}[f(\theta)]\). Then:
Initial alignment: When the prior over \(\pi_E\) is diffuse, the builder optimizes broadly, approximating performance on the full universe \(F\).
Posterior concentration leads to specialization: As \(\hat{\pi}_{E,t}\) concentrates around \(\pi_E\): \[ \lim_{t \to \infty} \theta^*_t = \arg\max_{\theta \in \Theta} \mathbb{E}_{f \sim \pi_E}[f(\theta)] \] If \(\pi_E\) is non-uniform, the builder specializes to high-density regions at the expense of broad capability.
This is the core tension: privatization works in one-shot settings, but repeated evaluation reveals information that enables strategic specialization. The builder cannot identify a fixed evaluation set (since \(F_E^{(t)}\) is resampled each round), but from repeated observations the builder learns \(\pi_E\) itself — the distribution from which evaluation tasks are drawn.
11.5.3.3 The Pareto Frontier
Define the residual misalignment at time \(t\) as the evaluator’s regret from the builder optimizing for its (drifting) belief rather than for \(u_E\) directly: \[ \Delta_t := \max_{\theta \in \Theta} u_E(\theta) - u_E(\theta_t). \]
This measures the gap between the best model society could get and the model the builder actually fields once it has begun to learn \(\pi_E\). When the builder’s effective belief equals \(\mu\) — in particular when \(\pi_E = \mu\) so that what leaks is exactly the construct — \(\Delta_t = 0\).
No evaluation strategy can simultaneously achieve low variance and low leakage. Variance decreases in \(k\); leakage increases in \(k\). Varying \(k\) traces a Pareto frontier where the evaluator must choose their preferred operating point.
Figure. The information-variance Pareto frontier across \(k \in \{5, 10, 20, 50, 100\}\) over 200 rounds with \(|F| = 200\) tasks. Left: evaluation noise (score variance) is lower for larger \(k\). Center: the builder learns \(\pi_E\) faster with larger \(k\), as measured by KL divergence between the builder’s posterior and the true \(\pi_E\). Right: as the builder’s posterior concentrates, incentive misalignment grows. No single \(k\) achieves both low noise and low misalignment — a fundamental tradeoff.
11.5.4 Restoring Alignment
The Pareto frontier of the previous section seems like an impasse: any choice of \(k\) sacrifices either measurement precision or incentive alignment. The resolution comes from recognizing that the evaluator is also a learner. If the evaluator corrects their distribution over time, the information the builder accumulates becomes obsolete.
11.5.4.1 Distribution Correction
The right target for correction is the construct \(\mu\), not uniformity. The evaluator does not reweight blindly toward a flat distribution; it discovers which regions it under-covers and fills them.
Embed tasks in a feature space, so each \(f \in F\) has a location \(z_f\). At each round \(t\), the evaluator:
- Discovers a gap: samples a location \(\nu_t\) with probability proportional to the local deficit relative to the social-relevance distribution \(\mu\), \[ P(\nu_t = f \mid \pi_E^{(t-1)}) \;\propto\; \bigl[\mu(f) - \pi_E^{(t-1)}(f)\bigr]_+ . \]
- Fills locally: mixes in a Gaussian bump centered at the gap, \[ \pi_E^{(t)} = (1-\rho)\,\pi_E^{(t-1)} + \rho\, g_{\sigma_c}(\cdot\,;\nu_t), \qquad g_{\sigma_c}(f;\nu) \propto \exp\!\bigl(-\|z_f - z_\nu\|^2 / 2\sigma_c^2\bigr), \] normalized over \(F\). The parameters are the correction rate \(\rho \in (0,1]\) and the correction bandwidth \(\sigma_c > 0\). The process drives \(\pi_E^{(t)} \to \mu\).
In practice, evaluators discover these gaps through external channels: incident reports from users who hit model failures the benchmark missed; systematic audits of coverage across capability dimensions (languages, modalities, reasoning types); A/B testing of rankings under different subsets; and improving acquisition capacity over time. Each channel fills an under-covered region rather than flattening the distribution — correction aims at \(\mu\), the construct, not at uniformity.
Under gap-targeted Gaussian correction, with \(m_t = k\cdot\min(t, 1/\rho)\) the builder’s effective sample size, the expected residual misalignment satisfies \[ \mathbb{E}[\Delta_t] \leq \min\!\left(\underbrace{\frac{m_t}{m_t + |F|}}_{\text{estimation-limited}},\;\; \underbrace{(1-\rho)^t \sqrt{0.5\,D_0} + \varepsilon(\rho, \sigma_c)}_{\text{correction-limited}}\right), \] where \(D_0 = \mathrm{KL}\!\left(\pi_E^{(0)} \,\|\, \mu\right)\) is the initial bias of the evaluator’s distribution from the construct, and \(\varepsilon(\rho,\sigma_c) = O\!\left(\sqrt{\rho}\cdot\operatorname{TV}(g_{\sigma_c}, \mu)\right)\) is fluctuation from the stochastic gap-discovery process (vanishing as \(\sigma_c \to \infty\) or \(\rho \to 0\)).
The two terms capture complementary regimes — a race between two learners: the builder learns \(\pi_E^{(t)}\) from observations while the evaluator corrects \(\pi_E^{(t)}\) toward \(\mu\).
- Estimation-limited (\(t\) small): \(m_t \approx kt\), so \(\Delta_t \lesssim kt/|F|\). The builder has too few observations to identify the bias.
- Correction-limited (\(t\) large): \(\Delta_t \lesssim (1-\rho)^t \sqrt{D_0/2} + \varepsilon\). The builder may know \(\pi_E^{(t)}\) well, but correction has driven it toward \(\mu\), leaving almost nothing to exploit.
The crossover is at roughly \(t^* \approx |F|/(k + |F|\rho)\): before \(t^*\) estimation limits the builder, after it correction does. Crucially, \(k\) appears only in the estimation term — a smaller \(k\) slows the builder’s learning, buying the evaluator time to correct.
Figure. Alignment recovery under gap-targeted Gaussian correction toward a non-uniform construct \(\mu\), with \(|F| = 200\). Left: residual misalignment \(\operatorname{TV}(\pi_E^{(t)}, \mu)\) falls toward zero for all \(\rho > 0\), faster for larger \(\rho\); the \(\rho = 0\) baseline (gray) stays biased. Right: the bias \(\operatorname{KL}(\pi_E^{(t)} \,\|\, \mu)\) decays on a log scale, settling at a small floor set by the correction bandwidth \(\sigma_c\) (the \(\varepsilon(\rho,\sigma_c)\) term of the theorem).
11.5.4.2 Optimal Evaluation Size
With distribution correction, the Pareto frontier from Section 11.5.3.3 collapses: each revealed task gives the builder only transient advantage, since the bias it reveals is corrected within \(O(1/\rho)\) rounds. In steady state the builder’s effective sample size saturates at \(m_\infty = k/\rho\) (new observations arrive at rate \(k\), old ones age out at rate \(\rho\)), so the estimation-limited term of the theorem above becomes a steady-state exploitation cost \(k/(\rho|F| + k)\). Combining it with the evaluation variance \(\sigma^2/k\) gives the asymptotic per-round loss \[ \mathcal{L}(k) = \underbrace{\frac{\sigma^2}{k}}_{\text{evaluation variance}} + \underbrace{\frac{k}{\rho|F| + k}}_{\text{steady-state exploitation}} . \]
The first term decreases in \(k\) (more tasks, less noise); the second increases in \(k\) (more leakage) and decreases in \(\rho\) (faster correction shortens the exploitation window).
If \(\rho|F| > \sigma^2\), the loss \(\mathcal{L}(k)\) is minimized at \[ k^* = \frac{\sigma\,\rho|F|}{\sqrt{\rho|F|} - \sigma}. \] If \(\rho|F| \leq \sigma^2\), \(\mathcal{L}\) is strictly decreasing in \(k\) and the optimum is the largest admissible \(k\) (variance reduction dominates uniformly).
In the high-correction regime \(\rho|F| \gg \sigma^2\), \(k^* \approx \sigma\sqrt{\rho|F|}\), which gives clean comparative statics:
- \(k^*\) grows with \(\sqrt{\rho}\): each revealed bias goes stale within \(\sim 1/\rho\) rounds, so faster correction permits a larger evaluation set.
- \(k^*\) grows with \(\sigma\): higher score noise demands more samples for a reliable signal.
- \(k^*\) grows with \(\sqrt{|F|}\): a richer task universe dilutes any single leak.
- The threshold \(\rho|F| > \sigma^2\) is itself informative: when correction reaches fewer new tasks per round than the noise scale, no interior optimum exists and the evaluator should simply maximize \(k\). The capacity to identify and correct biases, \(\rho\), is thus the key design lever — it both lifts this threshold and raises \(k^*\). Investing in correction (coverage audits, benchmark changelogs, incident-driven task additions) is the most effective move in benchmark design.
Figure. Asymptotic evaluator loss \(\mathcal{L}(k) = \sigma^2/k + k/(\rho|F| + k)\) for several correction rates, with stars marking the optimal \(k^* = \sigma\rho|F|/(\sqrt{\rho|F|} - \sigma)\). Faster correction (larger \(\rho\), hence larger \(\rho|F|\)) lowers the minimum loss and pushes \(k^*\) rightward, scaling as \(\sqrt{\rho|F|}\) once \(\rho|F| \gg \sigma^2\). Parameters: \(\sigma = 1\), \(|F| = 2000\).
Together, the two levers of this section — randomization to hide \(D\) and correction to keep \(D\) moving toward \(\mu\) — make hiding the distribution durable: leaked information about \(D\) goes stale faster than the builder can exploit it. What they cannot address is the case where \(D\) itself is the wrong target, i.e., \(D \neq \mathcal{C}\). That gap is the subject of the next section.
11.6 Publishing the Construct: The Construct-Validity Protocol
Hiding \(D\) buys durability, but it raises two practical worries. If the distribution is secret, how does anyone — a regulator, an adopter, the builder itself — know the score means anything? And how is a builder supposed to improve against a target it cannot see? The construct-validity protocol resolves both by exploiting the very gap that caused the trouble. Because a single construct \(\mathcal{C}\) admits many operationalizing distributions, the evaluator can keep one distribution private while making others public.
The evaluator:
- Publishes the construct \(\mathcal{C}\) — a precise statement of the capability being measured (its content domain and boundaries), but not a benchmark.
- Keeps \(D\) and \(S\) private — the operationalizing distribution and every realized set stay secret, evaluated only inside a controlled pipeline.
- Publishes one or more proxy distributions \(D', D''\) — public benchmarks that operationalize the same \(\mathcal{C}\) (think GSM8K as a public proxy for a private competition-math distribution). Builders train and self-evaluate against the proxies.
The protocol rests on construct validity (Chapter 1): if \(D\) and \(D'\) genuinely operationalize the same construct, then capability transfers between them — a model that improves on \(D'\) because it acquired the construct should also improve on the private \(D\). This gives the evaluator a falsifiable check that requires no exposure of \(D\): compare a model’s private score to what its public-proxy performance predicts. Construct-valid improvement (real capability) transfers; proxy-specific gaming does not, and reveals itself as a model that beats the proxy yet underperforms its predicted private score. The private benchmark thus functions as a held-out construct probe, while all public optimization pressure lands on the proxies.
This is why \(\mathcal{C}\) need not — and should not — be private. Publishing the construct is what makes the score interpretable (validity) and what gives builders a legitimate target to improve against; privacy is reserved for the operationalization, where it buys durability. The protocol is the constructive use of the \(\mathcal{C} \supsetneq D\) gap: the same multiplicity of operationalizations that lets a builder mistake \(D\) for \(\mathcal{C}\) also lets an evaluator certify \(\mathcal{C}\) through public proxies while never revealing \(D\).
The protocol is an aspirational design rather than a closed theory. Making “transfer across operationalizations” a precise, testable guarantee is itself a predictive-evaluation question — whether public-proxy performance predicts held-out performance — and connects the strategic results of this chapter to the predictive-modeling tools developed earlier in the book. It is an active direction for the field, and a natural foundation for a privacy-preserving evaluation challenge.
11.7 Metric Design as Principal-Agent Problem
We now shift from which tasks to show (the object axis of Section 11.2) to which metrics to report — an orthogonal, reporting-side axis. Even with a perfectly designed evaluation mechanism, the choice of metric can create perverse incentives.
11.7.1 When Metrics Create Perverse Incentives
A vivid illustration comes from healthcare. In 2001, the New York State Department of Health began publishing hospital mortality rates for cardiac surgery. Dranove et al. (2003) documented the consequences: hospitals began avoiding severely ill patients rather than improving care. By publishing average treated outcome (ATO) as the quality metric, the system rewarded patient selection rather than treatment effectiveness.
The AI evaluation analogy is direct. A leaderboard that rewards average benchmark score incentivizes developers to specialize on easy benchmarks or to select favorable evaluation conditions, rather than improving broadly. The metric itself creates the misalignment.
Formalizing this as a principal-agent problem (Laffont and Tirole 1986): the principal (evaluator) chooses a reward function \(w\), and the agent (developer) best-responds with policy \(\pi^w\). The principal’s regret is \(R(\pi^w) = \max_{\tilde{\pi} \in \Pi} V(\tilde{\pi}) - V(\pi^w)\), where \(V(\pi) = \mathbb{E}[Y(\pi) - Y(0)]\) is the total treatment effect.
The average treated outcome \(w_{\text{ATO}}(x, t, y) = y \cdot \mathbf{1}[t = 1]\) can have unbounded regret. An agent maximizing \(\mathbb{E}[w_{\text{ATO}}]\) may achieve \(R(\pi^{w_{\text{ATO}}}) = \max_\pi V(\pi)\) — the worst possible policy.
The intuition is stark: a hospital maximizing average treated outcome selects only the healthiest patients for treatment, achieving excellent ATO but zero (or negative) total treatment effect. In the AI setting, a developer maximizing average benchmark accuracy across their selected tasks may simply drop the hardest benchmarks from their evaluation suite.
The total treatment effect metric \(w_{\text{TT}}(x, t, y) = y - \hat{\mu}_0(x)\) achieves zero regret when the principal has unbiased counterfactual estimates \(\hat{\mu}_0(x) = \mathbb{E}[Y(0) \mid X = x]\).
The key insight: by subtracting the counterfactual baseline (what would have happened without treatment), the metric removes the incentive for patient selection. In AI evaluation, this translates to scoring the “capability uplift” — performance relative to a baseline — rather than raw performance. This connects to the doubly robust estimation from Chapter 9: counterfactual metrics require causal reasoning about what performance would have been absent the developer’s effort.
11.7.2 Information Asymmetry and Metric Elicitation
Beyond metric choice, there is a prior question: how much should the developer reveal about their model’s capabilities? Consider a developer (agent) with a private cost-correlated variable \(X\) — for instance, knowledge about which capability dimensions their model excels at. The evaluator (principal) can design contracts conditioned on \(X\) if it is revealed.
Drawing on the information elicitation framework of Wang et al. (2024), the key question is: when does the developer prefer to reveal, conceal, or garble information about their capabilities?
- Reveal: The developer discloses \(X\) fully. The principal designs a targeted contract, which benefits both parties when the information enables efficient allocation.
- Conceal: The developer hides \(X\). The principal must use a one-size-fits-all contract, which is robust but potentially inefficient.
- Garble: The developer discloses a noisy version of \(X\). This intermediate option can be Pareto-improving.
The principal’s expected utility is weakly higher under revelation than concealment: \(V_P^{\text{reveal}} \geq V_P^{\text{conceal}}\).
Proof sketch. Under revelation, the principal’s optimization problem has a strictly larger feasible set (contracts conditioned on \(X\)), so the optimum can only improve. \(\square\)
But the developer’s incentives are more nuanced. Revelation helps when conditioning on \(X\) sufficiently differentiates high-cost and low-cost types, allowing the principal to offer more efficient contracts. Concealment is preferred when revelation would lead the principal to extract all of the developer’s surplus through perfectly targeted contracts.
11.7.3 Garbling as Differential Privacy
The most interesting case is garbling: the developer reveals \(Y = X\) with probability \(\varepsilon\) and \(Y = \xi\) (noise) with probability \(1 - \varepsilon\). This is precisely the randomized response mechanism from differential privacy.
Under fairly wide conditions (e.g., two exponential cost types with different means), the agent may prefer garbled disclosure to both full concealment and full revelation. Moreover, garbling can increase total welfare compared to concealment.
The intuition is that garbling provides the principal with just enough information to design better contracts without enabling full surplus extraction. In the AI evaluation context, this maps to:
- Model cards with calibrated noise: Developers disclose approximate capability profiles rather than exact performance vectors.
- Differential privacy in metric reporting: Adding Laplace noise to reported metrics creates a garbling mechanism.
- Benchmark transparency tiers: Publish aggregate scores publicly, but release fine-grained breakdowns only through a privacy-preserving mechanism.
This connects directly to the holdout mechanism of Section 11.4.2: both use calibrated noise to create better incentive equilibria. The parallel is deep — the evaluator’s decision about how much to reveal about the evaluation distribution and the developer’s decision about how much to reveal about model capabilities are dual problems in information design.
Figure. Left: heatmap of the agent’s net benefit from revelation vs. concealment across pairs of exponential cost types \((\lambda_0, \lambda_1)\). Red regions indicate the agent prefers revelation; blue regions favor concealment. The black contour marks indifference. Right: agent utility under garbling (noisy disclosure with probability \(\varepsilon\)) for fixed cost types. An intermediate garbling level can outperform both full concealment and full revelation.
11.8 Representative Benchmark Selection
A distinct but related design question: given a large suite of evaluation metrics, how should one select a representative subset? This is the “lite benchmark” problem faced by BIG-bench (200+ metrics \(\to\) BIG-bench Lite, 24), HELM (\(\to\) HELM Lite), and Cal Hospital Compare (hundreds of quality measures \(\to\) 12). In the language of Section 11.2, a representative subset is one whose induced distribution best approximates \(D\) — and hence the construct \(\mathcal{C}\) — at a fraction of the evaluation cost.
11.8.1 The Subset Selection Problem
Let there be \(n\) metrics and \(m\) alternatives (models). Each metric \(i\) produces a ranking \(\sigma_i\) over the alternatives. We seek a subset \(K \subseteq N = [n]\) of metrics that is “representative” of the full set \(N\). But what does “representative” mean formally? The work of Procaccia et al. (2025) provides two precise definitions grounded in social choice theory.
11.8.2 Positional Representation
The first notion prevents under-representation at every rank cutoff.
A subset \(K\) satisfies positional representation for group size \(g\) if for every rank cutoff \(r \in [m]\) and every alternative \(a\): \[ C(K, r, a) \geq \left\lfloor \frac{C(N, r, a)}{g} \right\rfloor \] where \(C(S, r, a) = |\{i \in S : \sigma_i(a) \leq r\}|\) counts how many metrics in \(S\) rank alternative \(a\) in the top \(r\).
In words: if alternative \(a\) is ranked in the top \(r\) by at least \(\ell \cdot g\) metrics in \(N\), then \(a\) must be ranked in the top \(r\) by at least \(\ell\) metrics in \(K\). The parameter \(g\) controls the granularity of representation — smaller \(g\) requires finer representation but demands larger \(|K|\).
The minimum subset size needed to guarantee positional representation satisfies: \[ \Omega\!\left(\frac{n}{g} \cdot \frac{\log m}{\log(n \log m / g)}\right) \leq |K| \leq O\!\left(\frac{n}{g} \log m\right) \] The upper bound is achieved by a polynomial-time greedy algorithm (Algorithm 1 in Procaccia et al. (2025)) based on set cover.
The greedy algorithm works as follows: iterate through the preference profile row by row, coloring entries when an alternative accumulates \(g\) appearances. Then greedily select metrics that cover the most remaining colors. The connection to set cover yields the logarithmic factor.
11.8.3 Positional Proportionality
Positional representation prevents under-representation but not over-representation. A stronger notion prevents both:
A subset \(K\) satisfies \(\epsilon\)-positional proportionality if for every alternative \(a\) and every rank cutoff \(r\): \[ \left| \frac{C(N, r, a)}{|N|} - \frac{C(K, r, a)}{|K|} \right| \leq \epsilon \]
This preserves the fraction of metrics ranking each alternative at each position, up to additive error \(\epsilon\).
The minimum subset size for \(\epsilon\)-positional proportionality satisfies: \[ \Omega\!\left(\frac{1}{\epsilon^2} \log m\right) \leq |K| \leq O\!\left(\frac{1}{\epsilon^2} \log m\right) \] These bounds are tight up to constant factors.
The tight bounds follow from a connection to uniform convergence: a random subset of size \(O(\epsilon^{-2} \log m)\) satisfies positional proportionality with high probability, by a Chernoff-type argument applied simultaneously to all \(O(m)\) alternatives and rank cutoffs.
A particularly useful consequence connects proportionality to scoring rules:
If \(K\) satisfies \(\epsilon\)-positional proportionality, then for any scoring rule with score vector \(s\) and every alternative \(a\): \[ |f_s(a, \sigma_N) - f_s(a, \sigma_K)| \leq \epsilon \] where \(f_s(a, \sigma_S) = \frac{1}{|S|} \sum_{i \in S} s_{\sigma_i(a)}\) is the average score of alternative \(a\) under scoring rule \(s\) in metric set \(S\).
This is the key practical guarantee: a subset satisfying positional proportionality approximates any scoring rule on the original set of metrics. The connection to Section 7.1 is direct: positional proportionality provides a formal criterion for when a “lite” benchmark preserves the same information as the full suite, complementing the dimensionality analysis of factor models.
Figure. Left: subset size \(|K|\) required by the greedy algorithm for positional representation at various group sizes \(g\), compared to random selection and the theoretical \(O(n/g \cdot \log m)\) bound. The greedy algorithm consistently outperforms random selection. Right: for \(g=5\), comparison of the fraction of metrics ranking each model in the top 5, between the full set (\(n=50\)) and the greedy-selected subset. The subset closely preserves the ranking structure.
11.9 Synthesis: Design Principles for Strategic Benchmarks
The three threads of this chapter — information design, metric design, and representative selection — converge on six actionable design principles for AI evaluation.
| Principle | Formal Basis | Recommendation |
|---|---|---|
| 1. Randomize and Refresh | Single-sample alignment (TV bound), Prop. (transparent failure) | Use randomized evaluation with task renewal. Transparent fixed benchmarks are Goodhart-vulnerable by construction. |
| 2. Correct and Grow | Gap-targeted correction theorem, Lemma (optimal \(k^*\)) | Invest in distribution correction (\(\rho\)) toward the construct \(\mu\) as the primary lever. The optimal \(k^* \approx \sigma\sqrt{\rho|F|}\) grows with \(\sqrt{\rho}\). |
| 3. Gate Information Release | Holdout mechanism, Prop. (learning slowdown) | Use threshold tests to condition information flow on builder behavior. Combine with distribution correction. |
| 4. Align Metrics with Welfare | Prop. (ATO regret), Prop. (TT zero regret) | Score total treatment effect, not averages. Account for counterfactual baselines. |
| 5. Allow Partial Transparency | Prop. (garbling dominance) | Calibrated noise in metric reporting (garbling) can create Pareto improvements over both full transparency and full opacity. |
| 6. Ensure Representative Subsets | Thms. (positional representation/proportionality bounds) | Use formal representation criteria rather than ad hoc selection when creating lite benchmarks. |
These principles interact in important ways. Principles 1–3 address the information channel (how much the evaluator reveals about the evaluation mechanism). Principle 4 addresses the metric channel (what the evaluator measures). Principle 5 addresses the developer channel (what the developer reveals about their model). Principle 6 addresses the scope channel (which metrics to include).
A well-designed evaluation system should operate on all four channels simultaneously: randomize task selection (Principle 1), invest in distribution correction (Principle 2), gate information via holdout mechanisms (Principle 3), use welfare-aligned metrics (Principle 4), allow developers to disclose through privacy-preserving mechanisms (Principle 5), and ensure the metric suite is formally representative (Principle 6).
11.9.1 Three Properties of a Durable Evaluator
The six principles above are mechanisms; it helps to name the properties they secure. Read through the object-first lens of Section 11.2, a well-designed evaluator should guarantee three things.
- Validity. The score carries its intended construct meaning: it tracks \(\mathcal{C}\), not merely the distribution \(D\) that operationalizes it. Validity is secured by publishing the construct and proxies (Section 11.6) and by welfare-aligned metrics (Section 11.7) that reward capability rather than its proxies.
- Durability. The benchmark stays informative as it is optimized against, because \(D\) stays private and is corrected toward \(\mu\) faster than it is learned (Section 11.5). Hiding the set (Section 11.4.2) buys time; hiding and correcting the distribution buys durability.
- Faithful execution. The evaluator runs exactly the announced procedure — the same sampling, scoring, and reporting for every subject, with no special access or after-the-fact adjustment. This single property subsumes what one might separately call neutrality (no subject is favored) and integrity (the promised procedure is the one that runs): both are guarantees about honest execution of a fixed protocol, and both fail in the same way — a strategic evaluator (Section 10.3.2) — which the builder-facing mechanisms of this chapter do not address.
The natural implementation of faithful execution is procedural: run sampling, evaluation, and reporting inside a secure, auditable pipeline — a sealed environment that draws \(S \sim D\), scores submissions, and emits only the agreed signal, while producing a verifiable log that the announced protocol was followed. Randomization protects the builder from the evaluator’s information; auditable execution protects every party from the evaluator’s discretion. Only the first is a mechanism-design question; the second is why Chapter 10 treats evaluator-side gaming as a matter of governance.
| Object hidden | Mechanism | Property secured |
|---|---|---|
| nothing | static benchmark | — (fails) |
| set \(S\) | reusable holdout / Ladder | durability (bounded) |
| distribution \(D\) | randomization + correction | durability |
| \(D, S\); publish \(\mathcal{C}, D'\) | construct-validity protocol | validity + durability |
| the protocol itself | secure auditable pipeline | faithful execution |
11.10 Discussion Questions
The evaluation game framework assumes a benevolent evaluator (social planner). What changes if the evaluator also has strategic incentives — for example, a company running its own benchmark to favor its own models?
How does the information-variance tradeoff relate to the reliability-validity tradeoff from Chapter 5 and Chapter 1? Is there a formal connection between \(\sigma^2/k\) (evaluation variance) and Cronbach’s \(\alpha\)?
Design a mechanism where the builder’s incentive is to improve on the hardest tasks rather than the average. How should \(u_E(\theta)\) and \(r(\theta, S)\) be modified?
In what sense is the Chatbot Arena a randomized evaluation mechanism? Does it satisfy the conditions of the single-sample alignment theorem (Section 11.5.1)? What information leaks through the adaptive matching process?
How should a government regulator think about the correction rate \(\rho\) and the transparency level \(k\) for a safety benchmark? Which is the more effective lever, and what factors (incident-reporting infrastructure, task-acquisition capacity, score-noise scale \(\sigma\)) should influence the choice?
If two “lite” benchmarks both satisfy \(\epsilon\)-positional proportionality with the same \(\epsilon\) but select different subsets, which should be preferred? What additional criteria might break ties?
The holdout mechanism draws on differential privacy. What is the relationship between the privacy budget \(\varepsilon\) in the holdout mechanism and the notion of \(\varepsilon\)-differential privacy from the data privacy literature?
Distribution correction requires the evaluator to “know their bias.” How can an evaluator identify which regions of the task universe they under-represent? What role do meta-evaluations and coverage audits play?
Pick three named benchmarks (e.g., FrontierMath, LiveBench, Chatbot Arena) and classify each by which object it keeps private — the construct \(\mathcal{C}\), the distribution \(D\), or only the set \(S\) (Section 11.2). Where the answer is “only \(S\),” what would it take to also hide \(D\)?
A dynamic benchmark refreshes its test set every round. Construct a scenario in which it remains exploitable even with zero item-level leakage. (Hint: consider \(D \neq \mathcal{C}\) and Section 11.4.3.) What does this say about the limits of contamination control?
Faithful execution (Section 11.9.1) is the one property the chapter’s builder-facing mechanisms do not secure. Give an example of an evaluator behavior that violates faithful execution but that no amount of randomization or distribution correction can prevent. What kind of intervention would address it?
11.11 Bibliographic Notes
The Stackelberg evaluation game, the single-sample alignment bound, and the information-variance tradeoff are developed in Truong et al. (2025) (“Strategic Evaluation: Incentivizing AI Capability Coverage with Private Benchmarks”). The gap-targeted distribution-correction mechanism and the optimal evaluation size \(k^*\) are from the same work, which also reports a semi-synthetic experiment on MMLU-Pro using a low-rank latent-factor model of task structure. The analysis of metric design as a principal-agent problem draws on Wang et al. (2024), which develops counterfactual quality metrics (Chapter 5) and information elicitation in agency games (Chapter 6). The positional representation and proportionality framework is from Procaccia et al. (2025).
Goodhart’s Law was articulated by Goodhart (1984) in the context of monetary policy. Manheim and Garrabrant (2018) formalize four variants. The connection to reward overoptimization in RLHF is explored by Gao et al. (2023).
Strategic classification — the study of agents who manipulate their features to achieve favorable classification outcomes — was formalized by Hardt et al. (2016). Perdomo et al. (2020) introduce performative prediction, where the model itself changes the data distribution. Braverman and Garg (2020) show that randomness is necessary for efficient classification under strategic behavior.
Bayesian persuasion (Kamenica and Gentzkow 2011) provides the information design framework connecting evaluator disclosure to builder behavior. Bergemann and Morris (2019) survey the broader information design literature. Contract theory (Laffont and Tirole 1986; Holmstrom and Milgrom 1991) provides the principal-agent foundations for metric design.
Differential privacy and adaptive data analysis (Dwork et al. 2015; Dwork and Roth 2014) underpin the holdout mechanism. Blum and Hardt (2015) apply reusable holdout techniques to machine learning competitions. The connection between DP and strategic robustness is developed in the holdout mechanism analysis.
What to hide: sets versus distributions. The object-first framing of Section 11.2 — separating the construct \(\mathcal{C}\), the operationalizing distribution \(D\), and the realized set \(S\) — organizes a range of mechanisms by what they keep private. Reusable holdouts and the Ladder hide the set and bound leakage (Blum and Hardt 2015; Dwork and Roth 2014). Dynamic benchmarks refresh the set to defeat contamination — DynaBench (Kiela et al. 2021), LiveBench (White et al. 2025), and Chatbot Arena (Chiang et al. 2024) — but, as Section 11.4.3 argues, do not hide the distribution. Hiding the distribution is the contribution of the strategic-evaluation game (Truong et al. 2025). The construct-validity protocol of Section 11.6, which publishes the construct and public proxies while keeping the operationalization private, builds on the validity theory of Chapter 1 and remains an open, partly aspirational direction.
Social choice and benchmarking connect to Zhang and Hardt (2024) (Arrow’s impossibility for benchmarks), Colombo et al. (2022) (Borda count for benchmark aggregation), and Rofin and Mikhailov (2023) (scoring rules for benchmark ranking). The committee selection literature, particularly justified representation (Aziz et al. 2017), provides the social choice foundations for positional representation.
11.12 Exercises
(Easy) Show that if the builder’s belief matches social relevance, \(\tilde{\pi}_E = \mu\), the single-sample alignment is exact (\(\Delta = 0\)) and extends to any sample size \(k\), not just \(k = 1\). Hint: show that for any \(k\)-subset mechanism drawn from a \(\mu\)-matching belief, the expected reward is proportional to \(u_E(\theta)\).
(Easy) Verify that, when \(\rho|F| > \sigma^2\), \(k^* = \sigma\rho|F|/(\sqrt{\rho|F|} - \sigma)\) minimizes \(\mathcal{L}(k) = \sigma^2/k + k/(\rho|F| + k)\). Show that \(k^* \approx \sigma\sqrt{\rho|F|}\) in the high-correction regime \(\rho|F| \gg \sigma^2\), and explain why no interior optimum exists when \(\rho|F| \le \sigma^2\).
(Medium) Derive the estimation-limited term \(m/(m+N)\) in the misalignment bound using a Dirichlet prior centered at \(\mu\) over distributions on \(F\) with \(m\) effective observations. Show that the posterior predictive is the shrinkage estimator \(\hat{\pi}_E = \frac{m}{m+N}\hat{p} + \frac{N}{m+N}\mu\) and that \(\text{TV}(\hat{\pi}_E, \mu) \leq m/(m+N)\).
(Medium) In the agency game with binary \(X\), suppose \(C \mid X=0 \sim \text{Exp}(\lambda_0)\) and \(C \mid X=1\) is zero-cost. Derive the condition on \(\lambda_0\) and the principal’s value \(b\) under which the agent prefers concealment to revelation.
(Medium) Show that any subset \(K\) satisfying \(\epsilon\)-positional proportionality approximates any scoring rule within \(\epsilon\). Hint: Write \(f_s(a, \sigma_S)\) as a weighted sum of cumulative counts \(C(S, r, a)/|S|\) and apply Abel summation.
(Hard) Starting from the single-sample best response \(\theta^M \in \arg\max_\theta \mathbb{E}_{f \sim \tilde{\pi}_E}[f(\theta)]\), derive the single-sample alignment bound \(\max_\theta u_E(\theta) - u_E(\theta^M) \le 4B\operatorname{TV}(\mu, \tilde{\pi}_E)\). Under what conditions on the builder’s belief \(\tilde{\pi}_E\) does the builder maximize \(u_E(\theta)\) exactly?
(Hard) Prove that the holdout mechanism slows the builder’s posterior concentration by a factor of \(e^{\varepsilon\eta}\). Hint: Compute the effective sample size \(m_t^{\text{holdout}} = n_0 + p_{\text{alarm}} \cdot k \cdot \min(t, \rho^{-1})\) and compare to \(m_t^{\text{baseline}}\).
(Hard, computational) Implement the greedy algorithm for positional representation on the HELM Lite data (or a synthetic dataset with \(n = 100\) metrics, \(m = 50\) models). Compare \(|K|\) to the existing lite benchmark subset for different values of the group size \(g\). At what \(g\) does the greedy algorithm match the size of the existing subset?
(Medium) Consider a dynamic benchmark that draws a fresh set \(S' \sim D\) each round with no item-level leakage, where \(D = \pi_E\) is biased away from the construct weighting \(\mu\). Suppose the builder knows \(D\) and best-responds with \(\theta^* = \arg\max_\theta \mathbb{E}_{f \sim D}[f(\theta)]\). Show that the residual misalignment \(\max_\theta u_E(\theta) - u_E(\theta^*)\) is bounded below by a quantity proportional to \(\operatorname{TV}(\mu, D)\), and conclude that no refresh rate drives it to zero. (This is the claim used in Section 11.4.3.)
(Easy) Classify each of the following by the object it keeps private (construct \(\mathcal{C}\), distribution \(D\), or set \(S\)): (a) a public benchmark with a hidden answer key; (b) a leaderboard backed by the Ladder algorithm; (c) a monthly-refreshed live benchmark; (d) a private benchmark released only as a published construct plus a public proxy. Which are durable against a builder who learns \(D\)?
(Hard) Design a concrete transfer test for the construct-validity protocol (Section 11.6): given a model’s scores on public proxies \(D', D''\) and its score on a private \(D\), specify a statistic that distinguishes genuine construct improvement from proxy-specific gaming. What assumption about the relationship between \(D'\) and \(D\) does your test require, and how would you check it?