Software Engineering

SWE-bench Verified

SWE-bench Verified tests whether coding agents can resolve real issues reported against open source software projects by submitting a code patch that the benchmark's automatic evaluation accepts as a fix.

500items

134subjects

MITlicense

software_engineeringdomain

agents_and_tool_usedomain

textmodality

item-level responses released

Saturation status: No

Full data on Hugging Face Back to the gallery

Paper and credit

2024

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (opens in a new tab)

Original source

Authors

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao +3 moreShow all

01Carlos E. Jimenez
02John Yang
03Alexander Wettig
04Shunyu Yao
05Kexin Pei
06Ofir Press
07Karthik Narasimhan

Citation

BibTeX citationShow

BibTeX

1@misc{jimenez2024swebenchlanguagemodelsresolve,
2  title={SWE-bench: Can Language Models Resolve Real-World GitHub Issues?},
3  author={Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan},
4  year={2024},
5  eprint={2310.06770},
6  archivePrefix={arXiv},
7  primaryClass={cs.CL},
8  url={https://arxiv.org/abs/2310.06770}
9}

Response matrix

Rasch analysis p = σ(θ − z)

67,000 responses, 80/20 split over cells · 134 subjects · 500 items

AUC train

0.946

AUC test

0.941

Loading session strips…

Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect