Software Engineering
SWE-bench Verified
Build the SWE-bench Verified response matrix from the experiments repo
500items
134subjects
MITlicense
software_engineeringdomain
agents_and_tool_usedomain
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 120251215_livesweagent_claude-opus-4-50.792
- 220251205_sonar-foundation-agent_claude-opus-4-50.792
- 320250928_trae_doubao_seed_code0.788
- 420251127_openhands_claude-opus-4-50.776
- 520251120_livesweagent_gemini-3-pro-preview0.774
- 620250902_atlassian-rovo-dev0.768
- 720250804_epam-ai-run-claude-4-sonnet0.768
- 820250819_ACoder0.764
- 920250901_warp0.756
- 1020250612_trae0.752
- 1120250731_harness_ai0.748
- 1220251103_sonar-foundation-agent_claude-sonnet-4-50.748
- 1320250720_Lingxi-v1.5_claude-4-sonnet-202505140.746
- 1420250915_JoyCode0.746
- 1520251015_Prometheus_v1.2.1_gpt50.744
- 1620250603_Refact_Agent_claude-4-sonnet0.744
- 1720251103_SalesforceAIResearch_SAGE_OpenHands0.738
- 1820250522_tools_claude-4-opus0.732
- 1920251021_SalesforceAIResearch_SAGE_bash_only0.73
- 2020250522_tools_claude-4-sonnet0.724
- 2120250807_openhands_gpt50.718
- 2220250715_qodo_command0.712
- 2320250929_Prometheus_v1.2_gpt50.712
- 2420251014_Lingxi_kimi_k20.712
- 2520250710_bloop0.712
- 2620250623_warp0.71
- 2720250611_moatless_claude-4-sonnet-202505140.708
- 2820250519_trae0.706
- 2920250610_augment_agent_v10.704
- 3020250515_Refact_Agent0.704
- 3120250524_openhands_claude_4_sonnet0.704
- 3220250519_devlo0.702
- 3320250430_zencoder_ai0.7
- 3420250805_openhands-Qwen3-Coder-480B-A35B-Instruct0.696
- 3520250516_cortexa_o30.682
- 3620250930_zai_glm4-60.682