Reasoning
ARC-AGI
ARC-AGI tests whether a model can infer the transformation rule behind a handful of example input and output grids, then produce exactly the right output grid for a new input.
520items
74subjects
Apache-2.0license
reasoningdomain
gridmodality
item-level responses released
Saturation status: Yes
Response matrix
plus leaderboard aggregate scores
Rasch analysis p = σ(θ − z + c)
35,659 responses, 80/20 split over cells · 74 subjects · 520 items · 2 conditions
1 of 74 subjects answered every item alike, so its θ is unbounded and sits pegged to the column edge.
AUC train
0.970
AUC test
0.965
Loading session strips…
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect