Software Engineering
Terminal-Bench
Terminal-Bench tests whether agents working in a command line environment can carry out real engineering tasks such as building software from source, fixing a broken script or analyzing a dataset, checked by the task's own test suite.
89items
192subjects
Apache-2.0license
software_engineeringdomain
textmodality
item-level responses released
Saturation status: Yes
Response matrix
Rasch analysis p = σ(θ − z + c)
80,924 responses, 80/20 split over cells · 192 subjects · 89 items · 161 conditions
22 of 192 subjects answered every item alike, so their θ is unbounded and sits pegged to the column edge.
AUC train
0.920
AUC test
0.921
Loading session strips…
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect