Skip to main content
AIMSAI Measurement Science

Software Engineering

SOAR ARC-AGI

SOAR on ARC-AGI: per-(model, ARC-train task) binary success of an LLM-driven evolutionary program-synthesis search, across the base model (generation 0) and SOAR self-improvement iterations. A response is 1 if the model produced a Python program solving the original ARC task, else 0.

400items
5subjects
100%observed
unknownlicense
reasoningdomain
software_engineeringdomain
textmodality
gridmodality
Original sourcePaperBuild script← All benchmarks

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

SOAR ARC-AGI response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Subjects

  1. 1Qwen2.5-Coder-32B-Instruct0.6206
  2. 2Qwen2.5-72B-Instruct0.6191
  3. 3Mistral-Large-Instruct-24070.5659
  4. 4Qwen2.5-Coder-14B-Instruct0.4655
  5. 5Qwen2.5-Coder-7B-Instruct0.3556
Full data on Hugging FaceBack to the gallery