Software Engineering
Multi-SWE-bench
Multi-SWE-bench: multilingual issue-resolving benchmark over 8 languages (Python, Java, TS, JS, Go, Rust, C, C++). Per-instance resolved/unresolved verdicts for 39+ agent x model submissions, mirrored into a binary matrix.
2,078items
82subjects
Apache-2.0license
software_engineeringdomain
agents_and_tool_usedomain
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1OpenHands+Claude-3.7-Sonnet0.5711
- 2CodeArts-Agent+CodeArts-GLM-5.10.5433
- 3Agentless+Gemini-2.5-Pro0.5213
- 4SWE-agent+Claude-3.7-Sonnet0.4978
- 5Agentless+OpenAI-o3-mini-high0.4926
- 6CodeArts-Agent+CodeArts-MiniMax-M2.50.4912
- 7Agentless+OpenAI-o10.4898
- 8Agentless+Claude-3.7-Sonnet0.4636
- 9OpenHands+Gemini-2.5-Pro0.458
- 10Agentless+Doubao-1.5-thinking0.4571
- 11Agentless+DeepSeek-R10.4518
- 12OpenHands+Claude-3.5-Sonnet(Oct)0.4362
- 13Agentless+Claude-3.5-Sonnet(Oct)0.4283
- 14SWE-agent+Gemini-2.5-Pro0.4212
- 15Agentless+DeepSeek-V30.4192
- 16SWE-agent+Claude-3.5-Sonnet(Oct)0.4133
- 17RepoRepair+Claude-3.5-Sonnet(Oct)0.4122
- 18Agentless+Doubao-1.5-pro0.3982
- 19SWE-agent+OpenAI-o10.3945
- 20InfCode+GPT-5.20.3906
- 21Agentless+Llama-4-Maverick0.3818
- 22SWE-agent+Doubao-1.5-thinking0.3696
- 23Agentless+GPT-4o-11200.3686
- 24SWE-agent+OpenAI-o3-mini-high0.3557
- 25OpenHands+DeepSeek-R10.3485
- 26iSWE+Agent0.3386
- 27iSWE-OpenModels0.3125
- 28OpenHands+DeepSeek-V30.3042
- 29Agentless+Qwen2.5-72B-Instruct0.3039
- 30SWE-agent+Doubao-1.5-pro0.2952
- 31OpenHands+GPT-4o-11200.2936
- 32OpenHands+Doubao-1.5-thinking0.2802
- 33MSWE-Agent+CodeArts-MiniMax-M2.50.28
- 34OpenHands+OpenAI-o3-mini-high0.2636
- 35SWE-agent+GPT-4o-11200.2568
- 36InfCode+GPT-50.2558