Stanford UniversityAI Measurement ScienceAIMS — AI Measurement Science

Software Engineering

LiveCodeBench

LiveCodeBench per-sample binary outcomes from submissions; leaderboard-only rows remain aggregate pass@1.

1,055items
72subjects
CC-BY-4.0license
software_engineeringdomain
textmodality
item-level responses released
Saturation status: No
Original sourcePaper← All benchmarks

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

This condition combination wasn’t evaluated. Try a different attack, category, or judge.
lowhighUnobserved

Scale: {0, 1}

Subjects

  1. 1O4-Mini (High)0.873
  2. 2Kimi-k1.6-IOI-high0.8602
  3. 3DeepSeek-R1-05280.8474
  4. 4O3 (High)0.8474
  5. 5Gemini-2.5-Pro0.8466
  6. 6O4-Mini (Medium)0.8445
  7. 7Gemini-2.5-Pro-06-050.8427
  8. 8O1-2024-12-17 (High)0.8318
  9. 9Gemini-2.5-Pro-05-060.8275
  10. 10Gemini-2.5-Pro-03-250.8152
  11. 11Kimi-k1.6-IOI0.8107
  12. 12OpenReasoning-Nemotron-32B0.8098
  13. 13EXAONE-4.0-32B0.8088
  14. 14Qwen3-235B-A22B0.8038
  15. 15XBai-o4-medium0.8009
  16. 16QwQ-Max-Preview0.8
  17. 17OpenCodeReasoning-Nemotron-1.1-32B0.7853
  18. 18O1-2024-12-17 (Med)0.783
  19. 19Grok-3-Mini (High)0.781
  20. 20DeepSeek-R1-Preview0.7781
  21. 21Llama-3_1-Nemotron-Ultra-253B-v10.7773
  22. 22O3-Mini-2025-01-31 (High)0.7773
  23. 23O4-Mini (Low)0.7744
  24. 24Gemini-2.5-Flash-05-200.7621
  25. 25O1-2024-12-17 (Low)0.7591
  26. 26O3-Mini-2025-01-31 (Med)0.7536
  27. 27Gemini-2.5-Flash-04-170.7507
  28. 28QwQ-32B_temp0.7346
  29. 29DeepCoder-14B-Preview0.733
  30. 30O3-Mini-2025-01-31 (Low)0.7062
  31. 31Claude-Opus-4 (Thinking)0.7043
  32. 32Claude-Sonnet-4 (Thinking)0.6853
  33. 33O1-Mini-2024-09-120.6841
  34. 34Llama-3_1-Nemotron-Nano-8B-v10.6443
  35. 35Claude-3.7-Sonnet0.6351
  36. 36DeepSeek-R1-Lite-Preview0.6339
Full data on Hugging FaceBack to the gallery