Skip to main content
AIMSAI Measurement Science

Software Engineering

RE-Bench

Per-(model x environment) binarized success (0/1) on METR RE-Bench, the suite of 7 ML research-engineering environments, from METR's publicly released Time Horizon run logs (score_binarized). Item content is the real full task instruction text shown to the agent.

7items
37subjects
97%observed
mitlicense
ml_engineeringdomain
agents_and_tool_usedomain
software_engineeringdomain
textmodality
Original sourcePaperBuild script← All benchmarks

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

RE-Bench response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Subjects

  1. 1Claude Opus 4.60.9744
  2. 2Claude Opus 4.50.6222
  3. 3Claude Sonnet 4.50.4524
  4. 4Grok 40.4416
  5. 5GPT-50.3824
  6. 6GPT-5.1-Codex-Max0.3509
  7. 7Gemini 3 Pro0.35
  8. 8GPT-5.20.35
  9. 9Claude 4.1 Opus0.2787
  10. 10GPT-5.3-Codex0.25
  11. 11o30.193
  12. 12Claude 4 Opus0.1905
  13. 13o4-mini0.1818
  14. 14gpt-oss-120b0.122
  15. 15Claude 4 Sonnet0.1111
  16. 16Claude 3.5 Sonnet (New)0.0921
  17. 17Claude 3.5 Sonnet (Old)0.0789
  18. 18Gemini 2.5 Pro Preview0.0727
  19. 19o10.0658
  20. 20DeepSeek-R10.0526
  21. 21Kimi K2 Thinking0.0488
  22. 22DeepSeek-R1-05280.0357
  23. 23Claude 3.7 Sonnet0.0337
  24. 24GPT-4 01250.0179
  25. 25GPT-4 11060.0133
  26. 26o1-preview0.0115
  27. 27GPT-4 Turbo0
  28. 28GPT-4 03140
  29. 29GPT-20
  30. 30Claude 3 Opus0
  31. 31DeepSeek-V30
  32. 32GPT-4o0
  33. 33Qwen2-72B0
  34. 34gpt-3.5-turbo-instruct0
  35. 35Qwen2.5-72B0
  36. 36DeepSeek-V3-03240
Full data on Hugging FaceBack to the gallery