Software Engineering
LiveCodeBench
LiveCodeBench per-sample binary outcomes from submissions; leaderboard-only rows remain aggregate pass@1.
1,055items
72subjects
CC-BY-4.0license
software_engineeringdomain
textmodality
item-level responses released
Saturation status: No
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
This condition combination wasn’t evaluated. Try a different attack, category, or judge.
lowhighUnobserved
Scale: {0, 1}
Subjects
- 1O4-Mini (High)0.873
- 2Kimi-k1.6-IOI-high0.8602
- 3DeepSeek-R1-05280.8474
- 4O3 (High)0.8474
- 5Gemini-2.5-Pro0.8466
- 6O4-Mini (Medium)0.8445
- 7Gemini-2.5-Pro-06-050.8427
- 8O1-2024-12-17 (High)0.8318
- 9Gemini-2.5-Pro-05-060.8275
- 10Gemini-2.5-Pro-03-250.8152
- 11Kimi-k1.6-IOI0.8107
- 12OpenReasoning-Nemotron-32B0.8098
- 13EXAONE-4.0-32B0.8088
- 14Qwen3-235B-A22B0.8038
- 15XBai-o4-medium0.8009
- 16QwQ-Max-Preview0.8
- 17OpenCodeReasoning-Nemotron-1.1-32B0.7853
- 18O1-2024-12-17 (Med)0.783
- 19Grok-3-Mini (High)0.781
- 20DeepSeek-R1-Preview0.7781
- 21Llama-3_1-Nemotron-Ultra-253B-v10.7773
- 22O3-Mini-2025-01-31 (High)0.7773
- 23O4-Mini (Low)0.7744
- 24Gemini-2.5-Flash-05-200.7621
- 25O1-2024-12-17 (Low)0.7591
- 26O3-Mini-2025-01-31 (Med)0.7536
- 27Gemini-2.5-Flash-04-170.7507
- 28QwQ-32B_temp0.7346
- 29DeepCoder-14B-Preview0.733
- 30O3-Mini-2025-01-31 (Low)0.7062
- 31Claude-Opus-4 (Thinking)0.7043
- 32Claude-Sonnet-4 (Thinking)0.6853
- 33O1-Mini-2024-09-120.6841
- 34Llama-3_1-Nemotron-Nano-8B-v10.6443
- 35Claude-3.7-Sonnet0.6351
- 36DeepSeek-R1-Lite-Preview0.6339