The Predictive AI Evaluation Challenge opened in April 2026, the first competition run by AIMS. Participants submit code that predicts model responses on hidden benchmark items from a sparse set of observations.
The task treats evaluation as prediction: rather than running every model on every item, can we infer the rest? It is a direct, public test of the measurement ideas the lab works on.
The competition runs through the summer, with a submission deadline in August and winners announced in September. Top participants are invited to present at the AI Measurement Science Workshop at COLM 2026.