Software Engineering
ResearchCodeBench
ResearchCodeBench tests whether models can implement a missing piece of a recent machine learning paper's code, given the paper itself and the surrounding file, with curated tests deciding whether the filled in code works.
212items
32subjects
MITlicense
software_engineeringdomain
ml_engineeringdomain
textmodality
item-level responses released
Saturation status: No
Response matrix
Rasch analysis p = σ(θ − z + c)
6,784 responses, 80/20 split over cells · 32 subjects · 212 items · 20 conditions
AUC train
0.949
AUC test
0.917
Loading session strips…
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect