AI Measurement Science: Toward Rigorous AI Evaluation

Treating AI measurement as a first-class scientific problem for frontier AI.

Submit a paper Call for Papers Schedule Accepted papers

Venue: Hilton San Francisco Union Square, USA · Workshop room 10: Franciscan A&B / Ballroom level
Date: October 9, 2026 · Co-located with COLM 2026
Format: One-day, in-person · 8:30–17:30 PDT

01Call for Papers

We are excited to announce the inaugural AI Measurement Science Workshop, co-located with COLM 2026 at the Hilton San Francisco Union Square. The workshop is a one-day, in-person event on October 9, 2026, bringing together researchers from machine learning, statistics, psychometrics, economics, and policy to develop a unified perspective on measurement for frontier AI systems.

As AI systems become increasingly embedded in real-world workflows, existing evaluation paradigms face fundamental limitations. Contemporary benchmarks typically assume i.i.d. samples, stationary concepts, and passive models. These assumptions are increasingly violated in modern systems, which interact with environments, adapt to feedback, and are optimized against known evaluation signals.

This workshop frames evaluation as a problem of AI measurement science. We focus on three tightly coupled challenges (measurement of interactive systems, measurement under strategic optimization, and measurement under non-stationarity) and aim to define common abstractions and open research directions for this emerging area.

Submissions from educational testing, psychometrics, and related fields are strongly encouraged.

Two submission tracks.

Research Track

We invite work across theory, methodology, and systems, including but not limited to the three tightly coupled challenges below.

Interactive Measurement

Evaluation of policies, long-horizon behavior, tool-using agents, and statistical validity when observations are non-i.i.d. and path-dependent.

Measurement under Strategic Optimization

Evaluation as a game between learner and evaluator; robustness to gaming and contamination; incentive-compatible, private, randomized, or adaptive leaderboards.

Measurement under Non-Stationarity

Covariate, label, and construct drift; adaptive test design, recalibration, and longitudinal monitoring of deployed systems.

Competition Track

We also invite write-ups from participants of the Predictive AI Evaluation Challenge. Papers should describe each team's approach to predicting model responses from sparse observations, with methods, results, and lessons learned. Submissions follow the same 4–8 page COLM format as the research track and are reviewed on a later timeline so that final competition results can be included. The Best Competition Paper award will be presented alongside the workshop's research awards.

Competition track deadline: August 15, 2026

View the competition

02Submit

Submission guidelines.

Short, focused contributions in COLM format. Review is double-blind.

Papers of 4–8 pages in COLM format, with unlimited references.
Submitted via OpenReview; double-blind review, with at least three reviews per paper.
Accepted papers are non-archival.
Eight contributed talks selected from accepted submissions.

Important dates.

All deadlines are in Anywhere on Earth (AoE) time.

Research track submission deadlineJune 23, 2026
Research track notificationJuly 24, 2026
Competition track submission deadlineAugust 15, 2026
Camera-ready dueSeptember 21, 2026
WorkshopOctober 9, 2026 (COLM 2026 workshops day)

Submissions open now on OpenReview. Deadline June 23, 2026.

03Speakers

Speakers and panelists from academia, industry, and policy.

Invited speakers

In schedule order

Daniel Kang

UIUC

Assistant professor working on agentic benchmarks, AI security evaluation, and ML deployment systems.

Serena Wang

Harvard / UBC

Postdoctoral researcher on long-term AI impacts, counterfactual metrics for social welfare, and robust distillation.

Elham Tabassi

Brookings

Led NIST's AI Risk Management Framework and served as chief technologist of the U.S. AI Safety Institute; named one of TIME's 100 most influential in AI.

Panelists

Alphabetical by last name

04Schedule

October 9, 2026. All times Pacific Daylight Time (PDT). Program subject to change.

Scroll horizontally to view all columns.

Workshop schedule for October 9, 2026. All times are Pacific Daylight Time (PDT).
Time (PDT)	Session / presentation	Speakers / participants
08:30 – 08:45	Opening remarks	Sang Truong
08:45 – 09:15	Invited talk	Daniel Kang
09:15 – 10:15		—
09:15 – 09:30	Chasing the Public Score: User Pressure and Evaluation Exploitation in Coding Agent Workflows	Hardy Chen
09:30 – 09:45	Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration	Eliya Habba
09:45 – 10:00	LLM-as-a-Verifier: A General-Purpose Verification Framework	Jacky Kwok
10:00 – 10:15	Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference	Joel Persson
10:15 – 10:45	Invited talk	Serena Wang
10:45 – 12:00	Poster session 1	—
12:00 – 13:00	Lunch break	—
13:00 – 14:00		—
13:00 – 13:15	DelusionEval: Measuring Delusion-Linked Behaviors in AI Chatbots	Jared Moore
13:15 – 13:30	Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark	Christopher Stewart
13:30 – 13:45	Safety Benchmarks as Measurement Instruments: A Crossed Evaluation of LLM Harm Benchmarks	Emaan Bilal Khan
13:45 – 14:00	Spiking the training data to correct for test set contamination	Johnny Wei & Jerry Li
14:00 – 14:30	Hot Takes on AI Evaluation Slides for Hot Takes on AI Evaluation (opens in a new tab)	Olawale Salaudeen
14:30 – 15:00	Invited talk	Elham Tabassi
15:00 – 16:15	Poster session 2	—
16:15 – 17:30	Panel discussion	Panelists: Valerie Chen, Deb Raji, Nikil Ravi, Hanna Wallach, Angelina Wang, Adina Williams