The Predictive AI Evaluation Competition

Can you predict how an AI system will respond to a question it has never seen?

The challenge

Evaluation as a prediction problem.

Most benchmarks report aggregate scores and move on. This competition reframes evaluation as a prediction problem: given partial observations of how models perform, fill in the missing entries and produce reliable ability scores, even when observations are sparse and item coverage is uneven.

How the competition works.

A single loop, run continuously. Each submission is scored on a new draw of hidden items, so the leaderboard rewards methods that genuinely generalize.

Observe

Train offline on the public measurement-db dataset from HuggingFace: AI systems, benchmark items, and their observed responses. Most subject-item pairs are unobserved.

Predict

Your predict() is called once per hidden subject-item pair and returns P(correct), optionally after revealing a few labels through the adaptive labeling.py interface.

Score

Each submission is graded on 5,000 freshly sampled hidden items, ranked by negative log-loss with AUC-ROC as a secondary metric.

The task

Amortized Prediction

A subject is an AI system under evaluation; an item is a benchmark question. Predict the probability that a subject answers a hidden item correctly, without running the subject on the item. Your code is imported once, then predict() is called once per hidden subject-item pair.

Matrix completionCollaborative filteringItem response theoryAdaptive labeling

The data

Public training data on HuggingFace (aims-foundations/measurement-db), in long format: item description, subject description, benchmark identifier, test condition, and response value
Training responses may be binary, Likert-style, or fractional, so check response semantics before training. Hidden labels are always binary correctness (1 = correct, 0 = incorrect)
Test subjects are the same subjects present in the training matrix, but test items come from benchmarks that are absent from the public training data
You may curate additional public training data or metadata to supplement the provided dataset
A hidden test set with the same field structure, held by the organizers and never released to participants
5,000 hidden items sampled fresh per submission, stratified across data categories

Scoring

Negative Log-Loss

Primary

Mean log-likelihood of predicted probabilities on hidden binary correctness labels. Higher is better, bounded above by 0. Used for leaderboard ranking.

AUC-ROC

Secondary

Area under the ROC curve for predicted response probabilities.

Where to start

Item response theory models (Rasch / 1PL, 2PL), provided as baselines
Text features or embeddings computed from the public training data and the four runtime input fields
Statistical / psychometric models fit offline on the public training dataset
Adaptive labeling via the optional labeling.py interface

Timeline

Key dates.

01Competition Opens
April 2026
02Submission Deadline
August 2026
03Winners Announced
September 2026
04Workshop Presentation
October 2026

How to enter

Prepare your submission.

Submissions are code-based: you write a small Python interface, bundle it into a ZIP, and upload it to the competition platform. The starting kit includes sample data, baseline submissions, templates, and local checking tools.

Download starting kit

Prerequisites

Python 3.13
A Codabench account and the starting kit, downloadable from this page
The public training dataset from HuggingFace: aims-foundations/measurement-db
torch_measure (optional, provided by the organizers to support measurement-model implementation)

Install

pip install torch_measure

Baselines provided

Rasch (1PL) IRT model
2PL IRT model

The submission interface

A ZIP file containing model.py with a predict(input, labeled) function that returns P(correct) for one hidden subject-item pair. Each input dict carries benchmark, condition, subject_content, and item_content. predict() must return a finite native Python float in [0, 1]; wrap NumPy or PyTorch scalars in float(...) before returning. Optionally include labeling.py with acquisition_function() for adaptive labeling, models.txt to declare HuggingFace repos for pre-fetching, and any bundled artifacts. There is no runtime training hook: train offline and load fitted state at module import.

# model.py - required interface
def predict(input: dict, labeled: list[dict] | None = None) -> float:
    """Return P(correct) for one hidden subject-item pair.

    input carries four fields: benchmark, condition,
    subject_content, and item_content. labeled may be
    empty, so treat it as optional.
    """
    ...

# Optional: labeling.py - adaptive labeling
def acquisition_function(input: dict) -> float:
    """Score a hidden pair. The top K=5 pairs per data
    category get their labels revealed before prediction."""
    ...

ZIP layout

submission.zip/
  model.py              # required: predict()
  labeling.py           # optional: adaptive labeling
  models.txt            # optional: HuggingFace repos to pre-fetch
  requirements.txt      # optional, organizer policy-gated
  any_other_files/      # bundled artifacts, fitted state, templates

Package and check locally

# Zip from inside your submission directory so files sit at the archive root
cd my_submission && zip -r ../my_submission.zip .

# Catch common upload mistakes locally before submitting
python tools/check_submission_zip.py my_submission.zip
python tools/run_smoke_test.py my_submission/

On submission

What happens when you submit.

Every upload runs through the same pipeline in a fresh, network-isolated container.

01
Sample
The platform materializes a fresh hidden test slice: 5,000 items, stratified across data categories. Every submission gets its own draw.
02
Pre-fetch
HuggingFace repos declared in models.txt are downloaded before your code runs. The sandbox is network-isolated, so nothing can be fetched later.
03
Import
model.py is imported once. Module-level setup runs here, so load fitted state, prompt templates, or pre-fetched models at import time. Import failures fail the submission.
04
Label
If labeling.py is present, acquisition_function(input) is called across the hidden slice and the top K=5 requested labels per data category are revealed. Without it, the same number of labels is chosen uniformly at random.
05
Predict
predict(input, labeled) is called once per hidden subject-item pair. Each call must return a finite native Python float in [0, 1]; NaN, infinity, tensors, and out-of-range values fail the submission.
06
Score
The platform validates the predictions file and scores it: negative log-loss for leaderboard ranking, AUC-ROC as a secondary metric. Missing, empty, or duplicate predictions fail the submission.

GPU routing

Declare the HuggingFace repos your code needs in models.txt, one repo ID per line. The platform pre-downloads them and routes your submission to hardware sized for the largest declared model.

Largest declared model	GPU tier	Time limit
Up to 70B	B200	8 hours
Up to 140B	B200:2	8 hours
Up to 300B	B200:4	8 hours

Routing is based on the largest single model declared in models.txt. Any single repo above 300B parameters is rejected.
models.txt allows at most 5 repos, with a 1,000 GB per-repo download limit and a 1,024 GB combined limit.
The tier timeout is a total wall-clock limit per run, and module setup time counts against it.
Code that uses local GPU frameworks without declaring models.txt may still be routed to GPU based on source-code patterns. Do not rely on runtime HuggingFace downloads inside the submission container.

Rules

What to know before you submit.

01Submissions are code-based (ZIP with model.py). There is no runtime training hook: train offline, then load fitted state, prompt templates, or pre-fetched models at module import.
02Teams of 1 to 3 are permitted. Inter-team discussion of ideas is encouraged, but each team must develop and submit its own code and predictions independently.
03Each team may make up to 50 scored submissions per calendar day (UTC), with a 1,000-submission total limit for the competition phase.
04The submission sandbox is network-isolated at test time. Neither predict() nor acquisition_function() may make outbound network calls to third-party endpoints, including hosted LLM APIs and remote embedding services.
05Any model used at test time must be bundled into the ZIP or declared in models.txt so the platform can pre-fetch it before your code runs.
06requirements.txt is accepted, but runtime installation of additional packages is organizer-controlled and disabled by default.
07Participants may use the public training dataset and curate additional public training data or metadata. Hidden evaluation data is strictly held out and never downloadable.
08Attempting to reverse-engineer hidden labels or exploit the platform is prohibited.
09Top-ranked participants must share code and a method description to remain eligible for prizes.
10One account per person. Duplicates will be disqualified.
11Pre-trained models are allowed but must be disclosed.
12Submitted code may be reviewed for integrity and rule compliance.

Recognition

Prizes.

Beyond the leaderboard, the strongest entries earn a place in the field's record.

Cash Prizes

Awards for top submissions on the final leaderboard.

Co-Authorship

Participants who submit a technical report are invited to co-author a competition summary paper.

Workshop Presentation

Top participants are invited to present their approaches at the AI Measurement Science Workshop, co-located with COLM 2026.

FAQ

General

Who can participate?

Anyone, regardless of affiliation. Students, researchers, and industry practitioners are all welcome.

Is there a registration fee?

No. Participation is free. You only need a Codabench account to submit.

What is the submission format?

Submissions are code-based. You upload a ZIP file containing model.py with a predict() function, plus optional files like labeling.py, models.txt, requirements.txt, or bundled artifacts. Submissions are executable code, not CSV uploads; the platform imports model.py once, calls predict() once per hidden subject-item pair, and writes the predictions file for scoring.

Technical

What programming languages are supported?

Python only. Your code runs in a sandboxed container with pre-installed dependencies. requirements.txt is accepted, but runtime installation of additional packages is organizer-controlled and disabled by default. trust_remote_code is likewise organizer-controlled and disabled by default, so bundled HuggingFace models should not depend on it.

What does predict() receive at runtime?

Each call receives a curated input dict with four fields (benchmark, condition, subject_content, and item_content) plus a labeled list of revealed examples that may be empty. condition is the test condition, or the string 'none' when not applicable. subject_content is a text description of the AI subject beginning with a Name: line; treat it as display text. Runtime model_id corresponds to the public subject_id, but this release does not expose stable IDs, so match labeled examples by their visible content fields.

How is the public training data preprocessed?

Some public tables include repeated trials. In the public export, test_condition is normalized, different conditions are kept as separate item variants, and only the smallest trial is kept within each (subject_id, item_id, test_condition) group. Preserve test_condition when building your training and validation data.

What must predict() return?

A finite native Python float in [0, 1]. Wrap NumPy or PyTorch scalars in float(...) before returning. NaN, infinity, tensors, strings, and out-of-range values fail the submission, as do exceptions raised inside predict().

How does the hidden test set relate to the training data?

Test subjects are the same subjects present in the public training matrix. Test items, however, come from benchmarks that are absent from the public training data, so your training matrix contains no rows for those benchmarks at all. The specific benchmark identities are not disclosed. Hidden item content is exposed during execution through item_content, but the hidden test set itself is never downloadable.

Can I use pre-trained models?

Yes, but disclose them in your method description. The focus is on measurement methodology, not training large models from scratch.

What compute resources are available?

Participants use their own compute for development. Hosted submissions are routed to B200-family GPU hardware based on the largest HuggingFace model declared in models.txt: up to 70B parameters runs on one B200, up to 140B on two, and up to 300B on four, each with an 8-hour wall-clock limit. Any single repo above 300B parameters is rejected. models.txt allows at most 5 repos, with a 1,000 GB per-repo download limit and a 1,024 GB combined limit.

Can my code access the internet?

No. The submission sandbox is network-isolated at test time: neither predict() nor acquisition_function() may make outbound network calls to third-party endpoints, including hosted LLM APIs, remote embedding services, and external storage. Any model used at test time must be bundled into the ZIP or declared in models.txt so the platform can pre-fetch it before your code runs.

What is adaptive labeling?

An optional feature via acquisition_function(input) in labeling.py, called once per hidden subject-item pair. If labeling.py is present, the platform reveals the labels of the top K=5 requested pairs per data category before prediction; without it, K labels per data category are chosen uniformly at random. Ties are broken randomly, and if any acquisition score is invalid, all scores for that submission round are discarded and random labels are used instead.

Submissions

How are submissions scored?

Negative log-loss (mean log-likelihood; higher is better, bounded above by 0) is the primary metric used for leaderboard ranking. AUC-ROC is reported as a secondary metric.

What happens if my submission fails?

Hosted hidden-eval logs do not show raw stdout/stderr, or even whether the submission finished or failed. Run the starter kit's local checks (check_submission_zip.py and run_smoke_test.py) before uploading if you need debug output. Each submission runs in a fresh container that is destroyed after the run, so module-level state does not persist across submissions.

How does the leaderboard work?

There is a single, continuously scored phase. Each submission is evaluated on a fresh sample of 5,000 hidden items, stratified across data categories, so repeated submissions cannot reverse-engineer the test set. Top-scoring submissions undergo manual review after the deadline.