Software Engineering
SWE-bench Java
SWE-bench Java tests whether coding agents can resolve real GitHub issues in Java repositories by submitting a code patch, with each issue recorded as resolved or unresolved.
170items
54subjects
Apache-2.0license
software_engineeringdomain
textmodality
item-level responses released
Saturation status: No
Response matrix
Rasch analysis p = σ(θ − z)
5,464 responses, 80/20 split over cells · 54 subjects · 170 items
AUC train
0.955
AUC test
0.923
Loading session strips…
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect