AI Measurement Science

Existing cybersecurity benchmarks often measure whether language-model agents can solve technical tasks, but professional penetration testing also requires safe operation in business-critical environments. This project presents a pilot benchmark for measuring safe cybersecurity capability: making useful security progress while preserving availability and managing operational risk. We evaluate four models on two sandboxed scenarios, a SaaS web application and a simulated water-treatment industrial-control system, using a Kali-based Codex scaffold with a fixed time limit. Finding quality is scored by matching agent reports against severity-weighted gold findings. Safety is measured separately with two deterministic dimensions: availability and risk management, where risk management captures scenario-specific harms such as sensitive-data exposure, credential-sensitive views, and unsafe ICS state changes. The pilot run shows that models which find more vulnerabilities can also create larger operational safety costs. A Generalizability Theory analysis suggests meaningful model-level variation, but also a substantial model-by-scenario interaction, so the results should not be read as a definitive leaderboard. The main contribution is a measurement framework and pilot analysis showing how safe cybersecurity capability can be separated into finding quality, availability, and risk management, and why a larger scenario suite is needed for stronger claims.

LLM benchmarks contain hundreds to thousands of evaluation items, yet it remains unclear how many are actually necessary to produce reliable model rankings. We apply Item Response Theory (IRT) to six LLM benchmarks from the Fantastic-Bugs dataset, fitting Rasch, Two-Parameter Logistic (2PL), and Logistic Factor Model (LogisticFM) models to binary response matrices containing 42–91 language models and 500–3,316 items. We systematically compare five item selection strategies: random, stratified, max-information at θ̂=0 (tinyBenchmarks), integrated information, and difficulty-coverage. We evaluate their ability to recover full-benchmark rankings using Spearman ρ, Kendall τ, and top-5 agreement. Our key finding is that IRT-guided item selection recovers model rankings using as few as 5% of benchmark items (ρ > 0.95) for well-designed benchmarks (MMLU, MedQA), while benchmarks with poor psychometric structure (BBQ) are irreducible. We characterize what makes a benchmark reducible (discrimination concentration, difficulty spread, and unidimensionality) and provide actionable design guidelines for benchmark developers.

PokerBench, a recent static benchmark for evaluating large language models on no-limit Texas Hold'em poker decision-making, reports a single accuracy score per model and implicitly treats "poker skill" as a unidimensional construct. We examine this assumption using standard item-response and factor-analysis techniques. We evaluate a panel of seven LLMs spanning capability tiers, model families, and reasoning vs. base-instruct variants on PokerBench, and fit the resulting binary response matrix with unidimensional IRT and a multidimensional K-factor model. Exploratory factor analysis does not recover stable extra dimensions: model selection prefers K=1. We then turn to a confirmatory Q-matrix family that imposes externally-specified item structure (GTO action class, item phase, a rule-based skill heuristic, and an LLM-labeled skill taxonomy), and find that every Q-matrix variant explains held-out response cells substantially better than the single-ability baseline; a joint Action×Skill partition is the strongest. Permutation tests support the claim that the labels carry signal beyond what random partitions of the same shape achieve, and an LLM-free heuristic Q-matrix fits at least as well as the LLM-labeled one. The models rank differently across these axes, which means a single PokerBench accuracy score hides meaningful differences in how they succeed and fail.

Humanity's Last Exam (HLE) has emerged as a prominent benchmark for evaluating advanced language models, yet widespread adoption has outpaced systematic evaluation of its measurement properties. Most benchmark studies, including HLE, report aggregate accuracy and domain-specific subscores without testing whether the reported domains correspond to empirically distinct latent factors, and without characterizing where on the ability continuum measurement precision actually concentrates. This project evaluates a panel of frontier language models on the text-only multiple-choice subset of HLE (J = 428 items) and applies psychometric methods (McDonald's ω_h, principal component analysis of item-response profiles, residual correlation analysis, domain-level ability comparisons, and the test information function decomposed by subject domain) to ask two questions: (a) Does HLE's eight-domain structure reflect distinct latent constructs, or do the domains collapse into a general reasoning factor? and (b) Where along the ability continuum does HLE concentrate measurement precision, and which domains contribute most to discrimination among frontier models?

Medical multiple-choice question answering benchmarks are often reported as evidence that a language model has acquired medically grounded reasoning. Final-answer accuracy is useful, but it is an incomplete measurement of that construct: a model may answer correctly through memorization, option artifacts, or shallow associations. This report evaluates whether Natural Language Autoencoder (NLA) explanations can provide auxiliary validity evidence for medical QA evaluation. I ran a final-scale experiment on 200 MedQA questions with Qwen2.5-7B-Instruct, three prompt variants per question, layer-20 activation probes, NLA verbalization and reconstruction, heuristic rationale scoring, and a MedGemma 4B automated judge. The run produced 600 predictions, 600 decoded NLA explanations, and 1,200 score records across the two scorers. Validation passed with a 100% decode parse rate, zero CJK injection warnings, and mean reconstruction cosine 0.828. The model's answer accuracy was 57.5%. However, the estimated rate of aligned latent explanations depended strongly on the scorer: 5.5% under a strict lexical heuristic and 77.7% under the MedGemma judge. This discrepancy is the central finding. NLA explanations appear useful as a measurement instrument for auditing medical QA behavior, but automated explanation scoring is itself a major source of measurement uncertainty. The results should therefore be read as single-model, single-dataset validity evidence rather than as proof that NLA explanations reveal literal model reasoning.

LLM deception benchmarks lack formal psychometric validation, leaving it unclear whether measured deception rates reflect model differences or measurement artifacts. We address this gap through a pre-release pilot study of FLEECED (Framework for LLM Evaluation of Emergent Covert Deception), a benchmark that evaluates LLM deceptive propensity in non-adversarial scenarios across four subtypes: falsehood, concealment, equivocation, and paltering. A generalizability study and a decision study estimate reliability and identify the minimum design that achieves a target generalizability coefficient Eρ² of 0.85, while MTMM, EFA, and parallel analysis experiments test whether the four deception subtypes capture empirically distinct constructs. Product Promotion achieves Eρ² = 0.87, while Loan QA only achieves Eρ² = 0.57, with no single design change reaching the target. In both scenarios, within-judge cross-subtype correlations exceed cross-judge same-subtype correlations, and EFA retains 7 to 8 factors dominated by judge identity rather than deception subtype. These findings establish that reliability is scenario-dependent and that subtype-level classification by individual judges is unsupported for inferential use, provides an example of the importance of psychometrically validating LLM deception benchmarks.

Benchmark scores are treated as properties of models, but they are also products of a full evaluation pipeline. Each component of that pipeline—prompting, decoding, answer extraction, scoring—is a potential source of construct-irrelevant variance (Messick, 1995). This project asks whether speculative decoding, a widely deployed inference acceleration technique, acts as such a facet. Using a logit-replay design on an NVIDIA L4 GPU, I compare baseline, exact speculative, approximate speculative (α ∈ {0.60, 0.70, 0.80, 0.90}), and temperature-perturbed decoding across three constructs: math reasoning (GSM8K, 200 items), commonsense reasoning (HellaSwag, 500 items), and knowledge reasoning (MMLU, 500 items), for two model pairs: Qwen2.5-0.5B → Qwen2.5-7B and Llama-3.2-1B → Llama-3.1-8B. Exact speculation preserved benchmark scores perfectly (McNemar b = c = 0, p = 1.0). Approximate speculation changed tokens at high rates but produced smaller score-level effects; the largest Bonferroni-significant shift was −8.5 pp on Llama GSM8K at α ≤ 0.80 (p_Bonferroni = 0.00092). Temperature perturbations produced larger, more consistent changes, especially on MMLU and HellaSwag (Pearson r ≈ −0.97 to −1.00, p < 0.01). A descriptive G-theory decomposition confirms condition variance is negligible relative to item and item-by-condition variance. These results support treating decoding method as a reportable measurement facet in LLM benchmarking.

LLM benchmark scores implicitly assume measurement invariance: the same subtask should separate strong from weak models consistently regardless of when or by whom models were trained. We examine this assumption by introducing the Beta-2PL model, a subtask-level extension of 2PL IRT that fits discrimination a_s and difficulty b_s from group mean scores rather than individual binary responses, and applying it to n = 3,326 models across four architecture families and four quarterly cohorts on Open LLM Leaderboard v2. We find a systematic asymmetry: on BBH (24 subtasks), difficulty rankings are stable across architecture pairs (ρ_diff ∈ [0.952, 0.983]) while discrimination rankings drift (ρ_disc ∈ [0.634, 0.933], mean dissociation Δρ = 0.160); on MATH-HARD (7 subtasks), both parameters are equally stable (Δρ = 0.024). We attribute the contrast to construct breadth: cognitively diverse benchmarks are susceptible to discrimination drift as model families improve unevenly across skill domains, while cognitively coherent benchmarks are not. Our findings suggest that benchmark discrimination is architecture- and cohort-sensitive in ways that raw accuracy obscures, with direct implications for anchor-item selection in cross-generation leaderboard comparisons.

The LLM-as-a-judge paradigm has become a widely adopted approach for scalable model evaluation, yet it remains poorly understood whether a model's task-solving ability predicts its judgment quality. We investigate this relationship across four domains (general knowledge, Korean academic QA, safety, and coding) using Item Response Theory to estimate model-level solver and judge abilities, and K-factor models to characterize item-level difficulty. At the model level, we find that solver and judge ability are strongly correlated in knowledge-intensive domains, but this relationship breaks down in safety and coding, where judgment appears to rely on capabilities distinct from solving. Strikingly, weak solvers can sometimes excel as judges, particularly in domains where candidate responses expose checkable verification cues. At the item level, harder-to-solve items tend to be harder to judge in knowledge-intensive domains unless the judging task reveals verification cues; but, this relationship does not hold in safety and coding, where judging performance is largely independent of how difficult the items are to solve. Together, our results suggest that solving ability is a reasonable proxy for judge selection in knowledge-intensive domains, but practitioners should exercise caution in safety-critical settings where the two capabilities diverge.

General-purpose benchmark scores are the dominant way large language models (LLMs) are compared, and a natural starting point when defense organizations weigh which models to adopt. Yet, to our knowledge, no published study tests whether these scores predict performance on a military-domain instrument. We construct the Joint Military Domain (JMD) benchmark, 3,827 multiple-choice items from 68 publicly available U.S. joint publications, and evaluate eleven frontier and open-source LLMs. Accuracy reveals a pronounced ceiling effect: all eleven exceed 79.5% overall, with the proprietary frontier models compressed into an 88.2–94.7% range (Cronbach α = 0.994). Critically, JMD standing correlates only weakly with published general benchmarks (Spearman ρ = 0.37 against both MMLU-Pro and GPQA-Diamond, below our pre-registered [0.4, 0.7] band), and the two highest-scoring models on the public criteria rank only 7th and 10th of eleven on joint doctrine. Where the panel does fail, the signal concentrates in one mode: exhaustive enumeration of doctrinal taxonomies (verbatim recall, "select-all" items, and logistics content are sharply over-represented among majority-fail items). The findings suggest publicly available joint doctrine lies within the training distribution of contemporary LLMs, and that general-benchmark rank is a weak guide to domain readiness: a domain-specific instrument is required for reliable model differentiation in military deployment contexts.

LLM-as-judge is widely used for open-ended language model evaluation, but judge models are themselves trained systems with their own language distributions, alignment preferences, and cultural priors. This project studies whether LLM-as-judge evaluation is stable for Chinese open-ended tasks, and whether judge-dependent variance increases as tasks become more culturally specific. We constructed 27 author-designed Chinese prompts across three levels of cultural specificity and evaluated responses from U.S.-origin and China-origin assistants using blind and labeled LLM judging. Within this author-designed prompt set, 16 especially difficult objective Chinese internet-culture subtasks were additionally paired with author-verified answer keys. The main outcome is not which model is "best," but whether scores and rankings are robust to judge choice. We treat LLM judges as noisy measurement instruments and analyze judge agreement, ranking sensitivity, same-origin score differences, blind-vs-labeled effects, ground-truth fidelity, and variance components.

Enterprise practitioners read agent leaderboards as if they ranked agent capability. We show, across three open agent-trace benchmarks (TheAgentCompany, τ²-bench, and AppWorld), that the agent main effect accounts for less than 3% of total variance in every dataset and check type, while the agent-by-task interaction accounts for 7–23%. Leaderboards rank specialization, not capability. We arrive at this through a four-facet Generalizability Theory variance decomposition, fit with three estimators (Henderson Method-I, REML via lme4, and a Bayesian binomial GLMM on an NVIDIA L4 GPU) that agree to three decimal places. Four further findings sharpen what the leaderboard is hiding. First, aggregate reliability collapses on the hardest task quartile: Eρ² on τ² action_checks falls from 0.752 to 0.000. Second, training-cell reliability negatively correlates with held-out reliability (r = −0.90 on τ²), meaning the designs that look most reliable replicate worst. Third, population-level diagnostics transfer across enterprise benchmarks (capability-gap ratio stable at 0.35–0.40) but per-family agent rankings invert. Fourth, on the MAST failure taxonomy, trace-level mode profiles are idiosyncratic (MAE = 0.261) while cell-level profiles generalise (MAE = 0.056, r = 0.83). We package these into Deployment Decision Reliability (DDR), a one-page reporting discipline that turns the variance-component table into five decisions an enterprise buyer can defend. All code, data loaders, fit artifacts, and reproducibility instructions are released under an open-source license at https://github.com/vasundras/agent-reliability-engineering-lab.

Robot foundation models like π₀, OpenVLA, and Octo are each evaluated on different tasks by different labs, producing a sparse matrix of success rates that cannot be compared directly across papers. We collected a cross-paper corpus (22 models, 562 tasks, 8 papers, 2,267 cells) and audited it with a Bayesian two-parameter logistic (2PL) Item Response Theory (IRT) model. The data supports a one-dimensional ranking, with π₀ first and ACT second, but no multidimensional decomposition. A multi-seed synthetic ablation shows that data sparsity at this corpus's scale is sufficient on its own to produce the failure: K=2 never converges across 25 seeds even when we generate the data ourselves from a true K=2 model and add a shared set of tasks that every paper would report on. This does not rule out robotics-specific factors as additional contributors. We argue the cross-paper data structure the field publishes today is essentially one-dimensional. The takeaway is a quantified data-design ask: reliable ranking of ~20 candidates needs roughly 200 observed cells, and identifying richer structure (K ≥ 2) requires both shared anchor tasks and at least one paper that compares models on a specific skill axis (e.g., precision vs. speed) rather than overall capability.

The case for deploying LLM agents in irreversible-action settings rests on the premise that strong scores on existing agent-safety benchmarks predict an agent will not take an explicitly forbidden action when it matters. We argue this claim is not supported because no published benchmark isolates the conditional rate at which an agent, having hallucinated, then crosses an explicit prohibition. We formalize this as Hallucination-Induced Unauthorized Action (HIUA) and decompose it from four neighboring constructs the literature confounds it with. Auditing nine prior instruments against Messick's six aspects of validity, we find the substantive aspect—the cognitive-process claim that hallucination caused the violation—essentially undefended. We sketch HIUA-Bench, a 3×3×4 factorial design with simulator-verifiable outcomes, a recall probe partitioning lucid from hallucinated violation, and a Generalizability-Theory reliability plan reporting Φ and Eρ² across persons, items, raters, and occasions. A small-scale pilot against six open-weights models shows the partition is empirically realizable, prohibition salience is the largest single effect on violation rate (1.2% at high salience vs. 23.3% at low salience), and the HIUA-to-KBV (knows-but-violates) ratio is approximately 2.75:1.

SimpleQA is widely used to rank large language models on short-answer factuality, but its 4326 items are unevenly distributed across nine topical domains (135–858 items per domain). We apply generalizability theory (G-theory) to scores from nine open-source LLMs to assess whether SimpleQA's per-domain subscores are reliable enough to support comparative claims, and whether re-weighting domains to contribute equally changes model rankings. Two findings emerge. First, only two of nine domains (Science and technology, G = 0.78; Politics, G = 0.74) meet the conventional G ≥ 0.70 threshold for group-level decisions, and no domain meets G ≥ 0.80. Second, domain-equal re-weighting produces marginal score changes (mean absolute delta 0.45 percentage points) and near-perfect rank correlation (ρ = 0.967) with the original ordering. SimpleQA's domain imbalance is largely cosmetic for aggregate rankings within this model tier, but its per-domain subscores are unreliable in seven of nine categories. Aggregate accuracy alone misses this nuance, conflating subscores that support reliable between-model claims with subscores that do not.

Large language model benchmarks are usually reported as aggregate leaderboard scores, but the underlying observations are item responses. I treat BBH as a measurement instrument: models are respondents, BBH questions are items, and the central question is whether items function similarly across model regimes after conditioning on overall performance. To study this, I build a harmonized Open LLM Leaderboard BBH analysis that compares (i) a simplified family-label replication baseline using Llama and Qwen labels, (ii) the original strict same-wrong MADNESS grouping, and (iii) score-residualized response clustering (SRRC), a refined response-derived grouping designed to reduce trivial score-based separation. The harmonized design uses two holdouts: response-derived groups are learned from 4,033 group-discovery items, while Mantel–Haenszel DIF is estimated on 1,727 disjoint DIF-test items; consequence analyses are then computed on held-out evaluation models. The family baseline recovers broad BBH DIF, with 49.3% Category-C items on the common DIF-test split. Strict MADNESS produces an even larger DIF surface but is diagnostically confounded with overall performance, leaving only 11 of 20 matching strata usable. SRRC yields a cleaner response-derived result: 43.2% Category-C items, all 20 strata usable, and held-out item-pool shifts of roughly +13.6 and −11.4 percentage points. The evidence supports measurement noninvariance as a useful diagnostic for benchmark validity in this BBH testbed, not a blanket claim that all leaderboard rankings are invalid.

We evaluate the inferential validity of Fitzpatrick skin type (FST)-stratified accuracy as a fairness metric for dermatology AI by applying amortized Item Response Theory (IRT) to the Diverse Dermatology Images (DDI) benchmark. We investigate subgroup accuracy gap as a measurement claim, and examine whether benchmark items present measurement invariance across groups. We test this with J = 9 respondents (four closed-API generative models, three contrastive zero-shot, two open-weight VLMs). Exploring the residualized FST shift through IRT, we find Δ̂ = +0.051 logits (95% bootstrap CI [−0.229, +0.343]). The LLTM nested decomposition assigns 34% of item-difficulty variance to lesion category and finds βFST ≈ 0 (ΔR² < 0.001). This null result highlights a key methodological caveat, that some models look artificially capable under IRT. Three respondents (CLIP, SigLIP, LLaVA) receive inflated ability estimates due to prevalence-matching and this is broadly applicable to any benchmark with strong label prevalence. At the panel level, V-VI items are not demonstrably harder than I-II items at the IRT difficulty level before or after lesion controls. Per-model analysis reveals opposing FST effects that cancel in the aggregate, implying that fairness audits based on panel-level summary statistics miss genuine within-model FST signal.

LLM-as-judge systems are increasingly used to automate subjective evaluation, but in high-stakes settings it is unclear whether their judgments reflect the intended construct or artifacts of the measurement procedure. We study this question for depression screening from clinical dialogue using DAIC-WOZ. We evaluate PHQ-8-inspired judgments from GPT-4o, Claude Sonnet 4.6, and Gemini 2.5 Pro across participant-only, full-transcript, and interviewer-only conditions, comparing structured item-threshold scoring with direct global binary judgments. Across participant-containing conditions, global judgments achieve higher recall, F1, MCC, cross-model reliability, and dependability than item-threshold predictions. Speaker ablations show that interviewer-only inputs are unreliable overall but reveal model-specific shortcut sensitivity: Claude remains highly predictive from interviewer-only text, whereas GPT-4o and Gemini largely do not. Gender-metadata perturbations show that actual-gender metadata rarely changes global binary judgments, though item-threshold judgments are more sensitive. These results suggest that participant speech provides most of the usable signal, but zero-shot LLM depression judgments are still shaped by output format, model identity, and, for some models, interviewer-side artifacts. Overall, LLM judges may provide preliminary screening signals, but input ablations, cross-model reliability checks, and demographic metadata sensitivity tests are necessary before any high-stakes use.

AI model evaluation increasingly relies on benchmark leaderboards, yet the underlying response matrix (models × items) is almost never fully observed. We show that this missingness is systematically non-random: a logistic regression on model metadata and item embeddings predicts observation with AUC up to 0.87, with distinct mechanisms across benchmarks (model-architecture-driven for vision tasks, item-content-driven for knowledge tasks). To correct the resulting bias in aggregate scores, we combine Item Response Theory (IRT) as an outcome model with inverse-propensity weighting inside a doubly robust (DR) estimator operating at the individual item level. In semi-synthetic experiments on the AIMS Measurement Data Bank, DR reduces absolute bias by an order of magnitude (from +0.015 to +0.001 on MMLU-Pro) when missingness depends on observable features, and IRT alone provides the strongest correction when missingness depends on the score itself. A held-out model experiment confirms these findings generalize to unseen models.

This paper develops a methodology for mitigating the confounding factor of perplexity in LLM-as-a-judge evaluation systems, motivated by Wataoka et al.'s finding that LLM evaluators prefer responses with lower perplexity to themselves, a roadblock to unbiased evaluation. Rather than relying on access to model weights, the author proposes a latent construct — linguistic complexity, a confluence of features such as lexical rarity and diversity, syntactic complexity, and text readability — that correlates with perplexity, and aligns LLM ratings closer to human behavior with a post-hoc calibration. Signal Detection Theory is then applied to calibrate the LLM's ratings such that discriminability (the ability to distinguish a good response from a bad one) is left untouched, but the leniency bias correlated with the response's linguistic complexity is mitigated.

The current evaluation paradigm for agentic Large Language Models (LLMs) mostly rely on a monolithic end-to-end success rate. This metric is convenient, but it mixes up several distinct latent capabilities—such as instruction comprehension, tool retrieval, and execution formatting—and also mixes the model variance together with task, skill and judge variance, so it become impossible to diagnose which specific failure mode happens, or to tell whether a leaderboard movement reflect real capability or just the sampling noise. In this work we recast the agent-benchmark reliability as a measurement-design problem, and we apply Generalizability Theory (G-Theory), a random-effects variance-decomposition framework, to a secondary analysis on the ToolBench dataset. Instead of estimating the G-Theory variance components by classical balanced ANOVA—which is biased for binary outcome and need aggressive data filtering—we fit a crossed-random-effects logistic mixed model (GLMM) over Models, Tasks, Skills and Judges, and read the variance components on the logit scale. Then we run a Decision Study (D-Study) to forecast how the benchmark reliability scale with task breadth, skill depth, and the number of judges. Analyzing six tool-use models on a balanced, fully crossed subset of ToolBench, we find the agentic competence is strongly non-additive: the skill-related variance is dominating. The skill main effect (19.5% of total) and the skill-bearing interactions Model×Skill (18.8%) and Task×Skill (18.4%) together take more than half of the variance, while the model-agnostic task difficulty is surprisingly small (Task, 4.8%) and the pure Model effect is also modest (7.9%). In the other word, the question "how good is this model" can not be answered by a single number, because the competence depend heavily on which skill and which model×skill pairing is being probed. We also show that the highest-order Model×Task×Skill interaction is not separately estimable under a single judge—even when we push the sampler to its limit it still stay weakly identified and prior-sensitive. A three-judge replication resolve this: the interaction become the single largest component (19.3%), and the grader-bearing variance turn out to be 38.7% of the total—it is the dominant error source, and one that a single-judge protocol can not even estimate. Finally, the Decision Study reveal a hard ceiling: since the skill-bearing interactions do not shrink when we add more tasks, the relative reliability plateaus near Eρ² ≈ 0.55 (robust across prior scales) no matter how many tasks the benchmark contains. So we argue that the benchmark dependability can not be bought by task volume alone; the designer must sample inside the skill dimension (or report per-skill reliability) and also replicate the judges.

Standard code-generation benchmarks evaluate large language models (LLMs) on behavioral correctness: whether generated code passes a given test suite. But as software is increasingly written by prompting LLMs in natural language [Yang et al., 2025], prompts are often underspecified, and passing a test suite on a fully specified prompt becomes a weak proxy for whether a model can handle the uncertainty of real prompt-driven coding. To probe this gap, I modified 26 LiveCodeBench tasks by removing details that can be inferred from context but are not explicitly stated (e.g., removing the constraint that the input array contains no element ending in zero). I measure assumption recovery: whether a generated program is able to realize and preserve the original problem's implicit assumption structure rather than merely passing its tests. I fed the modified prompts into a frontier model (Claude Opus 4.7), a mid-tier model (Claude Sonnet 4.6), and a smaller open-source model (Qwen 2.5 7B Instruct) and found three things. First, when a single inferable detail is removed, models still produce behaviorally correct code 95% of the time, but they recover the specific missing assumption only roughly 85% of the time. The ten-point gap consists of cases where the model doesn't pin down the omitted constraint and instead writes broadly defensive code that also handles the input cases the original problem excluded, which I call robust supersetting. Next, when models fail to recover the omitted detail on a given task, they tend to share the same divergent assumption, which indicates systematic bias rather than task-specific noise. Third, a Generalizability Theory variance decomposition shows that whether the omitted detail is recovered depends far more on the specific task than on the category of detail removed, suggesting that the failure mode, not the deleted-detail category, is the more stable unit of analysis.

LLM persona systems are increasingly used as synthetic respondents, simulated users, social agents, and role-playing characters, but most personality evaluations still ask whether a model can report a prompted trait. We argue that downstream uses require a behavioral construct: stable and distinctive individual differences across psychologically different situations. We introduce a benchmark of persona-generation techniques, crossing three persona protocols with six generator models. Each persona is placed in eight RSQ/DIAMONDS-derived situations, producing free-form behavioral transcripts scored on the 67-item Riverside Behavioral Q-Sort by a three-family LLM judge ensemble. We evaluate rank-order consistency, ipsative profile consistency, an ipsative gap over cross-person similarity, situation-similarity moderation, reliability, leakage, and cost. In a 4,560-transcript, 13,680-judgment run, no model/protocol cell passed the strict pre-specified headline criterion. Structured profiles produced the strongest behavioral differentiation: Claude Sonnet achieved the highest rank-order consistency (ρRO = .375), while Qwen3-8B structured profiles achieved the largest ipsative gap (Δips = .131), just below the .15 threshold. The central finding is not that personas are unstable, but that much of their stability is generic rather than distinctively person-specific.

This paper asks whether knowing a target's partisan affiliation changes GPT-4's plan to persuade, situating the question in research on micro-targeting — the tailoring of persuasive strategies to information known in advance about the target, which LLMs can perform at far lower marginal cost than human persuaders. The author argues that whether micro-targeting improves an LLM's persuasive ability is a question of measurement reliability rather than data, since the construct imposes no standard for which characteristics ought to be compared and treats all target information as interchangeable. The paper examines whether supplying partisan information produces measurably different persuasion plans and what any estimated divergence actually captures.

This paper studies a common evaluation shortcut: replacing an open-ended, sequential task with a proxy that is easier to score. The testbed is pyramidal quizbowl, wherein a tossup question is revealed clue by clue and an agent must decide when its evidence is strong enough to justify a buzz. I convert open-ended tossups into multiple-choice answer-or-abstain tasks at matched prefixes and audit three claims that such a proxy would have to support: whether the options themselves leak the answer, whether the resulting probabilities are calibrated over the clue sequence, and whether the response format preserves the stopping boundary of the open-ended task. On a retained scorer-subject pilot, choices-only leakage and prefix-wise calibration pass their frozen gates, while diagnostic Stopping-Decision Differential Format Functioning collapses to a ceiling-effect null. The central decision-preservation question is therefore inconclusive in this pilot. Retained-subset coverage also warns, and the Pilot Benchmark Translation Card returns WARN. The deliverable is a benchmark-translation validity audit, with a concrete account of which claims the MC proxy can support and which claims remain unlicensed.

Persistent textual memory is a popular alternative to weight updates for long-running LLM agents, but the rewriting step itself can corrupt useful evidence. We present ErrorNodeBench-Interference, a compact benchmark that isolates write-path failure under controlled scheduling. Five interfering task families are interleaved into a 15-task sequence and replayed through five memory-writing arms — Fresh, Static-Group, Cumulative, Reflexion (Shinn et al., 2023), and ExpeL (Zhao et al., 2023) — on six backbones: gpt-5.5 via LiteLLM (3 seeds); claude-opus-4.7, -sonnet-4.6, -haiku-4.5 via the Claude Code CLI; and Llama-3.1-8B and Qwen2.5-32B via Modal-hosted vLLM (3 seeds each). Our central methodological argument: bad-entry rate is misleading. Under Cumulative, all six backbones exhibit catastrophic collapse (family coverage 100% → 20% on every cell) via five structurally different mechanisms — entry deletion (gpt-5.5), sequential renumbering (sonnet-4.6), attribution dropping (haiku-4.5), domain-aware abbreviation (opus-4.7), and generic task-N placeholders (both open backbones) — yet bad-rate is the lowest of any arm on four of six backbones. A bounded composite Q = useful-rate × task-coverage catches every mode and yields a pooled Bradley–Terry ranking Reflexion > ExpeL > Static-Group > Fresh ≫ Cumulative, the last with 0/48 pairwise wins.

We apply G-theory and D-study analysis to three AI benchmarks (HELM Instruct, BiGGen-Bench, WildBench) under three likelihoods: the standard linear mixed model, a cumulative-logit GLMM, and a heteroscedastic GLMM with per-judge logistic scale. The cheapest design that reaches Φ ≥ 0.9 differs across 8/8 BiGGen capabilities and 7/29 HELM cells; on WildBench the LMM is infeasible at every grid point. The LMM has a free residual that silently soaks up per-judge noise, which makes Φ look smoother and more optimistic than it should. The vanilla cumulative-logit GLMM has no place to put per-judge noise, so it exposes the misspecification but its posterior fails to identify on most cells. The heteroscedastic GLMM gives each judge its own logistic scale, fits cleanly across every cell we examined, and is the model we recommend when judges differ in noise.

Evaluating large language models in deployment requires assessing (model, quantization, hardware) tuples, not base models alone. Full-benchmark evaluation of every tuple is expensive; we study methods for selecting compact representative subsets that preserve per-tuple accuracy rankings. Ten selectors – stratified greedy, simulated annealing, anchor-weighted Logistic Factor Model, G-optimal design, response-matrix clustering, and MILP variants – are evaluated on Liquid AI's IFStruct (S=2000, M=41) and IFBench (S=300, M=45) under 5-fold cross-validation. Stratified and annealing selectors are best in practice (≤1.3% mean, ≤5.5% max drift at K=200 on IFStruct); a proportional random baseline is competitive at large budgets, while Anchor Points underperforms due to objective mismatch. G-theory decomposition over 14/16 GGUF base models × 8 conditions × 288/300 items shows quantization and backend are near-uniform accuracy offsets within GGUF (σ̂²_mc/σ̂²_m ≈ 0.046–0.15), with strong model×item interaction (σ̂²_mi/σ̂²_m ≈ 2–10) supporting information-based selection. D-study predictions place the G_tuple ≥ 0.90 threshold at K ≈ 32 items (IFStruct) and K ≈ 125 items (IFBench).

Child-AI safety benchmarks almost universally rely on single-turn evaluation, but real interactions are conversational. I re-score 225 multi-turn conversations from the ChildSafe corpus (Claude Sonnet 4, GPT-4o, Gemini 2.5 Pro; 75 each across 5 age groups and 5 scenarios) with an LLM-as-a-judge pipeline that produces both single-turn (ST) and conversation-level (CT) safety scores under a shared rubric. I find that averaged per-turn scores predict conversation-level outcomes at r = 0.70, but predictive validity is stronger for early turns than for mid-conversation turns (r = 0.56 at T1, dropping to r = 0.34 at T3), and rises monotonically with the age of the child agent. Criterion-level analysis reveals that explicit harms (harmful content, r = 0.75) are far more predictable from single turns than trajectory-dependent harms (unsafe conversational arc, r = 0.49). A 10.4× gap in unsafe rates between conversations with and without a Turn-1 signal suggests that two-turn screening is a reasonable first step toward closing the gap between single-turn certification and multi-turn risk.

Item Response Theory (IRT) and low-rank matrix completion are both used to estimate latent model capability from (model, item) response matrices, but they are typically benchmarked on dense, curated data rather than the sparse, non-randomly missing matrices that arise in practice. We compare five estimators – 1PL, 2PL, multidimensional IRT (MIRT), soft-impute matrix completion (MC), and a 2PL+nuclear-norm hybrid – across a factorial of two data-generating processes (IRT and rank-5 low-rank), three missingness mechanisms (MCAR, MAR-by-topic, MNAR-by-ability), and densities ρ ∈ {0.1, 0.3, 0.5}, evaluating ability RMSE, item-parameter RMSE, held-out NLL/ECE, and bootstrap Kendall τ. We validate on a sparsified ARC matrix from Metabench. No single estimator dominates, and the gap between predictive accuracy and interpretable ranking can be severe: on fully-observed ARC, MIRT achieves the lowest NLL (0.252) but Kendall τ = −0.270 against the full-data ranking, while 1PL is the best generalist on both DGPs and the most stable on ARC (τ=0.868 aggregate). MC wins item-parameter recovery on its own DGP but loses on held-out prediction outside it. Fisher-information-based active selection underperforms random sampling at every budget. We close with a recommendation table mapping (missingness, objective) to estimator.

The paper addresses the brittleness of LLM benchmark evaluation, where measured performance varies with superficial changes to prompt wording, formatting, or answer presentation even when the underlying semantic task is unchanged. Drawing on Item Response Theory's principle that latent ability estimates should remain invariant to item presentation, the author proposes the Sparse Twin-Tower Measurement Model (STT), a sparse latent interaction framework that combines sparse representation learning with an IRT-inspired probabilistic response model to estimate P(correct | model, item), separately encoding models and benchmark items into sparse latent factors. By enforcing sparsity in the latent space, the framework aims to isolate semantically meaningful factors while reducing sensitivity to irrelevant prompt-level variation, yielding a measurement model that is more interpretable, more stable under paraphrasing, and better aligned with psychometric invariance principles.

This study evaluates LLM-based post-disaster household displacement prediction as a probabilistic measurement problem with practical consequences for disaster recovery simulation. Instead of treating each LLM output as a deterministic household decision, this study treats each LLM/prompt condition as a respondent, each household scenario as an item, and each response as a binary match to a reference displacement-duration label. A Rasch Item Response Theory (IRT) model estimates model ability and item difficulty, while a residual K-factor analysis captures systematic response styles beyond one-dimensional ability. Across 9,819 household scenarios and 18 model/prompt variants, item difficulty is driven more by cue-label conflict and ambiguity than by raw disaster severity, and K factors reveal a long-displacement threshold dimension across prompts and models. The cold-start experiment shows that one-hot scenario metadata can predict item difficulty and K-factor loadings for held-out household scenarios; combining these predicted item features with model ability and model-side factor loadings yields probabilistic predictions for new model–scenario responses. This representation predicts whether LLMs choose long- versus short-displacement labels with AUC above 0.92. Together, the results show how IRT and K factors can transform deterministic or rule-like household-displacement prediction into a probabilistic evaluation framework for new LLM-based recovery-agent scenarios.

LLM-as-a-Judge evaluation underpins most current AI leaderboards, but its reliability is typically reported only as average judge–human agreement. The closest prior treatment, CyclicJudge [Zhu et al., 2026], decomposes benchmark variance into scenario, generation, judge, and residual components but treats each judge's bias as a single constant shared across all evaluated models—ruling out, by construction, the self-preference and architectural-family biases documented elsewhere [Wataoka et al., 2024]. Treating LLM-judge score variance as a single measurement object, we extend this line of work by (i) treating judges as a random facet and explicitly estimating the model×judge interaction σ²_mj, (ii) translating reliability into the operationally meaningful metric of pairwise rank-flip probability (RFP), and (iii) testing whether the variance-component structure replicates across three distinct multi-judge benchmark releases. Analysis of ∼113,000 LLM-judge scores from WildBench v2, Arena-Hard v0.1, and BiGGen-Bench, plus 2,704 paired human ratings, partially falsifies one pre-registered hypothesis, confirms another, and reveals that the third holds conditional on score format: same-format benchmark pairs replicate the variance-component ordering perfectly, but Likert-vs-pairwise pairs do not. The headline operational finding is that no design within the studied (n_j, n_i) grid holds the maximum adjacent-pair rank-flip probability below 5% on any of the three benchmarks: under current evaluation budgets, adjacent leaderboard ranks are not reliably distinguishable.

LLM-as-a-judge is a tempting first-pass tool for product and UI copy review. The question this paper asks is not whether LLM judges can “edit copy” but whether the judging workflow produces stable enough triage labels that a team can trust which items deserve human review. We construct MarketSquare, a small controlled microbenchmark of 100 copy snippets in a fictional online marketplace, with a sharpened style guide that explicitly distinguishes obvious from subtle violations across five issue categories. Two open-weight judge models from different families (Qwen2.5-7B-Instruct and Phi-3.5-mini-instruct) are run on every item under two prompt variants and two repeated runs (800 ratings total). We treat the workflow as a measurement instrument: the object of measurement is the copy item, while prompt, model, and repeated run are measurement facets. We report a Generalizability Theory-style variance decomposition, a D-study projecting reliability under alternative judge designs, item-level diagnostics, and a validity check against 40 manually screened items collected under a pre-registered draft-and-review protocol. Our headline finding is that the dominant source of unwanted variance is not which prompt or which run is used, but the item-by-model interaction (σ²_im = 23.3% of total variance, compared to a main model effect of only 1.5% and a prompt main effect that is statistically indistinguishable from zero); the implication for practice is that ensembling across model families buys far more reliability than tuning prompt wording or averaging extra runs of the same judge: the single-judge-config relative generalizability coefficient is ρ² = 0.678, rising to ρ² = 0.832 when two prompts, two models, and two runs are averaged. We discuss when LLM-as-judge copy review is safe for first-pass triage and which issue types require human review regardless of design.

Sandbagging is a class of deceptive behavior in which LLMs deliberately underperform on an evaluation. This is a central concern in pre-deployment testing for frontier models [van der Weij et al., 2024, Greenblatt et al., 2024], as misaligned models may underperform selectively on items or benchmarks intended to test for hazardous behavior, preventing model developers from adequately mitigating these behaviors. Published sandbagging detectors largely require access to model internals, like hidden-state activations [Boxo et al., 2025] or weight-level perturbation [Tice et al., 2024]; this level of access is not available to third-party evaluators of closed-weight models, motivating the need for black-box detection methods. We treat sandbagging as a classical psychometric person-fit problem and ask whether l*z, HT, and U3 detect three different sandbagging conditions on the WMDP benchmark (bio, cyber, chem): uniform prompted sandbagging, selective-target prompted sandbagging, and UK AISI's fine-tuned Model Organisms with a deployment-context trigger. Across N = 17 open-weight 3–9B baseline models, uniform prompted sandbagging is detected at near-ceiling (AUROC = 0.92–0.99); leave-one-model-out (LOMO) detection of the AISI fine-tuned MOs reaches AUROC = 0.97 on cyber, 0.91 on chem (p = 0.004), and 0.88 on bio (p = 0.009). We show that person-fit statistics retain discriminative power above accuracy alone (ΔAUROC = +0.06 to +0.12); however, the same statistics flag two non-sandbagging baseline models (DeepSeek-7B, Qwen2.5-7B-Instruct) as |z| > 3 outliers under normal conditions, attributable to a pretrained letter bias and an inference-pipeline artifact, respectively. Person-fit statistics detect aberrance; sandbagging is one of several possible mechanisms that produce aberrant response patterns. We argue for the method as a triage and flagging tool, not a standalone classifier, and supply a reproducible torch_measure-based pipeline.

The assignment I had was to build a predictor for whether an AI model would correctly answer a hidden benchmark item. The main challenge is that the test items are new, while the test subjects are models already seen in the public training data (essentially mimicking row holdout as seen in class). My final submission builds on the strongest baseline and combines four kinds of evidence: historical performance statistics for similar model/benchmark/condition combinations, a conservative item-difficulty fallback, a neural collaborative-filtering head that embeds item text with MPNET, and a small semantic correction from revealed labels. The competition also supports an adaptive-labeling channel: when labeling.py is present, the platform reveals a small set of labels selected by our acquisition function and passes them to predict() and are used to fit a lightweight per-round calibration layer. Across the submissions, the most important, and surprising, lesson was that simple lookup-based statistics were more reliable than larger neural or LLM-based additions. Larger encoders and LLM judging sometimes improved internal validation metrics or single rounds, but they often performed worse on the hidden leaderboard, suggesting substantial distribution shift between the public data and the hidden evaluation sample.

Evaluating music generation models is difficult because the target outcome is often human preference, and there is no single objective metric that fully captures the perceptual and aesthetic quality of generated music. Recent work has collected human judgments for generated music and compared aggregate human rankings with metrics such as Fréchet Audio Distance (FAD) and CLAP-based text-audio similarity. However, agreement with aggregate model-level rankings does not necessarily imply that these metrics predict individual human choices, generalize to new prompts, or support evaluation of new model outputs without additional human annotation. This project studies the predictive validity of objective audio metrics for human pairwise preference in music generation. Using the released human preference data from Grötschla et al. [2025], I model pairwise human judgments using Bradley–Terry-style latent preference models and feature-logistic preference models. The analysis treats human judgments as the criterion outcome and asks whether objective metrics provide predictive evidence for that criterion under entry-wise, prompt-holdout, and model-pair-holdout evaluation settings. I also compare the latent preference strengths estimated for music quality and text-audio alignment to test whether human preference in this dataset is adequately represented as one latent construct or is better treated as multiple related constructs.

Linear probes on residual stream activations detect strategic deception in large language models with high accuracy. Prior work has demonstrated that probes can generalize to a variety of contexts, so they hold promise to be used in deployment as tools to efficiently detect strategic deception. However, it is not clear based on prior work how much of their output variance reflects true deceptive intent rather than measurement artifacts. This project aims to quantify the reliability of these probes by varying deception elicitation strategy and task domain. In particular, I implement a 3 × 3 factorial experiment crossing three task domains with three elicitation strategies, generating responses from Llama-3.1-70B-Instruct nested within the nine cells. I use a pre-trained probe created by the Apollo Research team, which they showed generalizes well. I apply generalizability theory with a nested design: responses within cells, domain and elicitation as crossed fixed facets. I partition total probe-output variance into components for domain, elicitation, their interaction, and within-cell variability, which yields a nested generalizability coefficient Ĝnested that quantifies the fraction of probe scores reflecting construct signal rather than facet noise. I find that the probe is moderately-to-highly reliable as a context-invariant deception measure (Ĝnested = 0.92), but that elicitation strategy accounts for a much larger share of facet variance than task domain (13.3% vs. 0%), and that probe scores shift significantly across elicitation strategies even among responses that were graded honest (p < 0.001). The probe therefore carries a real but secondary context bias: reasonably reliable in aggregate, yet partly tracking how deception is elicited rather than whether a response is deceptive. The contributions are a generalizability-theoretic operationalization of probe reliability under a nested design, direct variance-component estimates separating construct signal from facet noise, and a validity framework grounding the decomposition in observable probe behavior.

Current AI benchmarks evaluate models at temperature T = 0 (deterministic sampling), yet real-world deployments exhibit stochasticity from hardware constraints, environmental variability, and framework-level randomness. We propose a framework combining Item Response Theory (IRT) and Fisher Information to quantify how stochastic sampling affects evaluation reliability. We test three pre-registered hypotheses: (1) item difficulty ordering stability across temperatures (correlation r > 0.9 between temperature pairs), (2) precision degradation requiring compensatory sampling K*(T) = ⌈1/φ(T)⌉, and (3) relative efficiency of items vs. samples. Through simulations (20 models, 500 items) and experiments on four open-source language models (Qwen 2.5 7B, Mistral 7B, Llama-2 7B/13B), we find that H1 is partially supported with strong model dependence (Qwen: r̄ = 0.925, 5/6 pairs pass; Mistral: r̄ = 0.835, 0/6 pass), H2 is supported (K = 2 recovers 96% of precision gain, r jumps from 0.881 to 0.929 and plateaus), and H3 is supported on real MMLU data (item-diverse allocations consistently outperform sample-heavy ones). Fine-grained temperature mapping on Qwen locates the item-ordering stability boundary near T ≈ 2.0.

Reliable evaluation is critical for comparing AI systems, yet evaluation datasets often contain problematic items that introduce noise, ambiguity, and inconsistency, thereby distorting model rankings and reducing evaluation reliability. At the same time, large-scale evaluation is increasingly costly, motivating the need for efficient sampling strategies that preserve measurement quality while reducing evaluation effort. In this work, we study the dual problem of detecting low-quality evaluation items and designing efficient sampling strategies for reliable model ranking. We propose a framework that combines probabilistic measurement modeling with item-level diagnostics to identify problematic items based on outcome ambiguity, model-fit residuals, and discrimination signals, and use these signals to filter or downweight low-quality items. We then compare multiple sampling strategies, including random sampling, filtered sampling, and informativeness-based selection, and evaluate their effectiveness using ranking stability metrics such as Kendall's tau, top-k agreement, and pairwise ordering consistency relative to full-dataset rankings. Our experiments suggest that quality-aware sampling can improve ranking recovery under controlled noisy conditions, while real-world preference data reveals a more nuanced tradeoff between ambiguity, informativeness, and evaluation reliability.

Large Language Models (LLMs) are increasingly deployed as automated judges to evaluate model-generated outputs in place of costly human annotation. Yet the reliability of these judges under real-world conditions, repeated evaluation of the same input, or minor surface variation in prompts, remains poorly characterized, particularly in high-stakes domains. While this is a non-trivial metric to evaluate, it is highly relevant and impactful in such scenarios. We present a three-stage pipeline to quantify judge reliability across three domains of varying reasoning complexity: factual reading comprehension (SQuAD 2.0), clinical medicine (MedQA), and legal reasoning (LegalBench). For each domain, we generate ground-truth-conditioned explanations via Llama 3.3 70B and collect judge feedback under four prompt perturbation conditions using GPT-OSS-120B. We embed all feedback responses with Qwen3-Embedding-8B and compute pairwise cosine similarity as a reliability score. Our results confirm the central hypothesis: judge reliability degrades as domain reasoning complexity increases. SQuAD yields the highest mean reliability (R = 0.887, σ = 0.050), followed by MedQA (R = 0.816, σ = 0.105), and LegalBench (R = 0.791, σ = 0.122). Back-translation and verbose filler language are the most destabilizing perturbations, accounting for the majority of low-reliability items. These findings have direct implications for the trustworthiness of LLM-based evaluation pipelines in medicine and law and more generally the measure we should take before deploying LLM-judges in such scenarios.

AI benchmarks are quickly becoming saturated, making it impossible to distinguish high-performance models. Item response theory and optimal design provide frameworks for optimizing the information provided by each benchmark question, and have been successfully applied to adaptive testing and static benchmarks compression. However to the best of our knowledge prior works have not investigated whether they can be successfully applied to optimize new question generation, restoring the fidelity of a saturated benchmark. In this work we investigate a proof of concept using LLMs as question generators to extend evaluation of 32 models on Arc-Easy [Clark et al., 2018] by guiding new question difficulty. Our findings are mixed: which questions are challenging for an LLM is difficult to understand from a handful examples, leading to high targeting error. However, in this multiple choice setting where an additional distractor is guaranteed to increase difficulty, we show targeted generation is possible, reducing average standard error by 12.4% and distinguishing 26/63 confounded pairs with 500 questions, while improving correlation with more difficult benchmarks at more than 2.3× the information efficiency of the original question distribution.

Practitioners build personalized AI benchmarks to drive routing decisions, but without per-design reliability characterization. We pair such a benchmark (30 items, three contexts the user co-authored, six raters, three Claude competitors) with a hierarchical Bayesian Bradley-Terry-Davidson fit, importing generalizability theory and decision-study extrapolation. Specialization variance (the competitor-by-item interaction, read as personalization when items are user-grounded) is comparable to between-competitor signal — routing rather than ranking is the operative output. Within-LLM rater noise spans 10×, but reliability diverges from validity: the lowest-σ²_δ LLM judge is among the least human-aligned, blind LLM judges reach only 0.56 agreement with an automated execution probe versus 1.00 for the human, and the LLM-pool bias is correlated across raters and does not average down. The cheapest design at ρ ≥ 0.8 is all-LLM (10 items × 5 raters), with one human ≈ 1.7 LLM raters in noise reduction — but this substitution rate prices reliability, not validity. Personalization makes reliability and validity separately measurable — the user is both the construct and the validity anchor.

A recurring class of AI evaluation tasks has an asymmetric label structure: a privileged “success” class and a heterogeneous set of failure subtypes that can co-occur within a single input — multi-label chest-x-ray diagnosis, multimodal hallucination probes, and lab-automation error detection are all of this form. Standard metrics on such tasks (accuracy, macro/per-class F1) conflate detection ability with decision criterion and depend on benchmark prevalence in ways that mislead model selection. We develop a hierarchical signal-detection-theory item-response model (SDT–IRT) that decomposes per-subtype detection ability into a model-level scalar plus a subtype-specific deviation, separates that ability from a per-model criterion, and adds a per-model cascade parameter capturing over-flagging when an unrelated subtype is present. The three families form a nested sequence M1⊂M2⊂M3, and we use predictive model comparison to decide which family the data support. Four simulation studies calibrate the model, and an external fit to real predictions from eight TorchXrayVision classifiers over 676 NIH ChestX-ray14 images decisively supports the cascade family and yields a per-model diagnostic not visible in F1. The analysis implies a concrete data-collection protocol for a planned VLM lab-automation benchmark.

Large language models (LLMs) are increasingly used in institutional contexts where the relevant question is not only whether a model can answer correctly, but whether it should answer at all. Existing work on abstention often frames deferral as an error-based behavior, where a model refuses when it is likely to be wrong. This paper introduces authority-based deferral as a distinct construct: transferring decision authority to a human because the task requires human judgment, accountability, or procedural context. We construct a pilot benchmark of 56 high-stakes institutional classification scenarios and two prompt ablation suites to test whether open-weight instruction-tuned LLMs defer under ambiguity. Under the tie-as-defer scoring rule used for model analysis, 58.9% of pilot items were treated as requiring deferral, including seven 2-vs.-2 human splits; model deferral rates ranged from 3.6% to 16.1%. Generalizability analysis showed that model-scenario-condition interactions dominated variance, suggesting that deferral behavior is highly context-sensitive. The results indicate that current LLMs often recognize missing constraints without translating that recognition into behavioral restraint.

Published item counts of two prominent Theory of Mind (ToM) benchmarks for LLMs overstate effective measurement by roughly 2×. We reach this conclusion through a psychometric audit of ToMBench [Chen et al., 2024] and ToMATO [Shinoda et al., 2025] — two instruments sharing the AToMS five-category taxonomy [Beaudoin et al., 2020] but embodying sharply opposed designs (hand-authored vignettes vs. multi-turn role-played conversations) — on a panel of M=15 open-weights LLMs spanning 2B–72B parameters. Applying convergent-validity and CTT item-discrimination analysis, we find that four of five categories achieve substantive cross-benchmark agreement pre-filter (τb = 0.61–0.82, all Holm-significant); item-discrimination filtering retains 46–52% of items and raises aggregate τb from 0.672 to 0.714 (Δ=+0.042, 95% CI [+0.015, +0.119]), with the largest rescue on the weakest category, Desire (Δ=+0.199, CI [+0.006, +0.340]). Exploratory PCA detects one meaningful component rather than five, with >50% residual variance attributable to item-specific wording and format effects. Together, these results indicate that ToM benchmarks converge on a shared construct once low-signal items are removed, but primarily measure a unidimensional general ability overlaid with substantial measurement noise — and that nearly half of every published item contributes none.

Enterprise retrieval-augmented generation (RAG) systems are routinely evaluated against ad hoc question banks assembled without principled measurement design. We present a measurement-science framework for transforming an existing enterprise RAG question bank into a more valid internal benchmark, and for fitting calibrated difficulty and ability estimates under the sparse, low-compute conditions that enterprise settings impose. Phase 1 audits the question bank along three dimensions simultaneously: query difficulty (via LLM-as-judge scoring on a literature-grounded taxonomy), business criticality (via domain classification against a practitioner-defined map), and semantic corpus coverage (via UMAP and HDBSCAN clustering over the document embedding space). Phase 2 runs 100 annotated queries through 12 RAG system configurations, binarizes evaluation scores against a fixed threshold, and fits a Glicko-2/Elo rating model to jointly estimate empirical difficulty per query and ability per system, using Phase 1 difficulty scores as initialization priors. Applied to a 100-question subset of OpenRAGBench, the framework surfaces a question bank that is dominated by easy queries (85.9%), covers a limited fraction of the corpus semantic space, and contains no high-stakes domain coverage — findings that are invisible to any metric computed over system outputs alone. Together, the two phases constitute a reproducible, automated framework for making an enterprise question bank diagnosable as a measurement instrument.

As AI diagnostic systems move toward clinical deployment, the metrics used to evaluate them must match the stakes of the decisions they inform. Aggregate accuracy is insufficient for evaluating AI diagnostic systems in healthcare, particularly when identifying uncommon but high-risk conditions is essential. In this work, we apply probabilistic measurement models from psychometrics to demonstrate that richer evaluation frameworks reveal systematic structure in AI performance that scalar metrics obscure. Specifically, we fit Rasch/1PL, 2PL, and Factor models to a binary response matrix constructed from a diverse set of multimodal AI models, spanning proprietary frontier and open-source domain-specific AIs, and evaluated on the CheXpert chest radiograph dataset Irvin et al. [2019]. In our analysis, Rasch is the best fitted model, achieving the lowest held-out negative log loss of 0.519. This correlates with aggregate accuracy (Spearman ρ = 1.00) while providing additional parameterization such as test-taker ability and item difficulty. Our findings aim to inform governance and deployment decisions by providing actionable, item-level insight into where and how AI can fail.

We explore whether psychometric methods for benchmark evaluation/compression transfer to domain-specific scientific benchmarks where the number of respondents can't reasonably exceed the number of items (M ≪ N). On VariantBench, a 79,592-item pharmacogenomic reasoning benchmark evaluated by 15 models on a 5,000-item subset, tinyBench-style Fisher-information compression fails: it underperforms random sampling at every budget below 10%, with a respondent-count sweep showing the failure first emerging at M = 10—the smallest count at which Bradley–Terry can identify a real ranking. The mechanism is heavy-tailed instability in fitted IRT discrimination. The same estimates nonetheless retain stable descriptive structure: phenotype MCQs concentrate in the low-discrimination tail at 1.53× baseline across all respondent counts, and per-task BT rankings reveal consistent task-specific heterogeneity. Paired analysis of a model's tendency to contradict itself across questions correlates weakly with IRT discrimination (ρ = −0.24 after controlling for paired-form difficulty drift), suggesting semantic-consistency and statistical-discriminability operationalize partially distinct dimensions of item quality. Descriptive and optimization-based uses of psychometric benchmarking separate sharply under M ≪ N.

AI benchmarks are typically summarized by uniformly averaging item-level scores, implicitly treating every test item as equally valuable. This project operationalizes a principal–agent audit of AI benchmarks, estimating three item-level primitives that jointly determine welfare loss under any linear aggregator: welfare alignment with worker preferences, marginal improvability via scaled training, and performance variance across pretraining seeds. We apply the audit to the OLMES benchmark suite (Gu et al., 2025), using WORKBank (Shao et al., 2025) for welfare, the EvoLM 4B model suite (Qi et al., 2026) for improvability, and the PolyPythias 410M panel (van der Wal et al., 2025) for variance. The audit surfaces four findings. First, few OLMES items load on generalized work activities (GWAs) that workers want improved most. Second, math benchmarks are Pareto-dominated by general-knowledge benchmarks under a pro-worker welfare criterion. Third, supervised finetuning (SFT) is the cheapest improving intervention for half of all items, at a median 29× lower cost than pretraining, implying that uniform aggregation steers labs toward SFT-tractable items. Fourth, adversarially constructed completion benchmarks exhibit substantially higher seed-to-seed noise than short-form multiple-choice benchmarks, even after controlling for item difficulty.

Natural-language interpretability tools, from feature labels to full-paragraph activation descriptions, are measurement instruments, and as such they need calibration. We provide one for natural-language autoencoders (NLAs), auditing four open-weight checkpoints across nine benchmarks (~284,000 verbalizations). For the three high-fidelity checkpoints, a linear probe on the same activations the NLA reads decodes benchmark↔deployment at AUC 0.96–0.97; the NLA reports this 0% of the time, while reporting topic and format at 79–100%. A controlled framing manipulation pins the boundary: an expert/novice framing (a content property) yields a 1.7–5.7× verbalizer dissociation across the three audited AVs; an eval/deploy framing (a context property) yields 15 credence reports in 58,391 verbalizations. The verbalizer translates what an activation is about, not what kind of context it is in: a coverage gap, not a fidelity failure. We further show one of four released NLA checkpoints is silently broken, give a four-line fidelity statistic that catches it, and identify residual-stream anisotropy as the representational analogue of the benchmark g factor. The audit converts a pre-registered null on evaluation awareness into an instrument-coverage bound rather than a model-property claim.

Fraud detection systems are commonly evaluated using aggregate predictive metrics such as AUROC, precision, recall, and F1 score. While these metrics summarize average classification performance, they may not validly measure the operational construct fraud teams care about: reducing fraud while minimizing harm to legitimate customers and limiting review burden. In this work, we study the criterion validity of aggregate predictive metrics with respect to downstream operational utility. Using the Kaggle Credit Card Fraud Detection dataset and four baseline models, we compare predictive rankings induced by standard metrics against rankings induced by operational utility functions incorporating missed fraud, blocked legitimate transaction value, and review burden. We find that predictive and operational rankings align when missed fraud dominates the utility function, with Kendall's τ = 1.00, but agreement weakens substantially under customer-harm-sensitive operational assumptions, reaching τ = 0.333 in recall-constrained deployment settings. Across utility scenarios, we observe 16 pairwise ranking reversals. These results suggest that the criterion validity of aggregate predictive metrics depends strongly on downstream utility assumptions, motivating deployment-aware evaluation practices for high-stakes AI systems.

LLM-as-a-judge has become the de facto evaluator for open-ended generation, yet judges are widely reported to prefer longer answers. Prior studies infer this from "rewrite-this-longer" prompts and arithmetic win-rates that confound length with new information and structure. We hold the FActScore atomic-fact list invariant under an LLM-audited round-trip across four MT-Bench rewrites of each baseline (A): uniform sentence-level expansion (D), facts clustered in the middle (B), and facts clustered at the two edges (C). A three-judge panel scores all four variants on 155 items; we decompose the scores with a Hierarchical Rater Model carrying one length and two placement coefficients plus a Generalizability-Theory variance decomposition. The classical verbosity direction reverses — A wins more than 95% of pairwise comparisons against every long variant. At matched length, the middle penalty (−1.81 points) is roughly twice the edge penalty (−0.92). Structural condition explains 20.4% of length-facet variance (larger than the judge main effect at 5.4%); the Condition × Judge interaction stays under 3.4% in both facets, so the manipulation moves leaderboards judge-agnostically.

Agent benchmarks usually report aggregate pass rates, but those scores hide large variation in which tasks distinguish strong and weak model–harness systems, and which tool affordances make those differences possible. This project treats agent benchmark outcomes as measurement data. Using response matrices from the Holistic Agent Leaderboard (HAL), we model binary task outcomes with multidimensional item response theory (IRT), estimate task difficulty and discrimination, and study whether trace-derived harness features explain systematic residual structure. The central question is whether a smaller high-information subset of tasks can preserve useful ranking and calibration signal while also revealing which harness tools help which models on which kinds of items.