Software Engineering
ResearchCodeBench
ResearchCodeBench: 212 coding challenges from 20 recent (2024-2025) ML papers. Given the paper plus context code with a blanked snippet, an LLM fills in the missing implementation, graded by curated tests. 32 LLMs evaluated; per-(model, snippet) pass/fail over 6784 observations.
212items
32subjects
MITlicense
software_engineeringdomain
ml_engineeringdomain
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1GEMINI_2_5_PRO_PREVIEW_05_060.6415
- 2O3_HIGH0.5943
- 3GEMINI_2_5_PRO_PREVIEW_03_250.5896
- 4OPENROUTER_O4_MINI_HIGH0.5849
- 5O3_MINI_HIGH0.5236
- 6CLAUDE_3_7_SONNET_2025_02_190.5189
- 7GPT_4_10.5
- 8CLAUDE_3_5_SONNET_2024_10_220.4858
- 9O1_HIGH0.4811
- 10DEEPSEEK_R10.4575
- 11GEMINI_2_5_FLASH_PREVIEW_04_170.4528
- 12GROK_3_BETA0.4292
- 13OPENROUTER_DEEPSEEK_CHAT_V3_03240.4245
- 14GPT_4_1_MINI0.4245
- 15GPT_4O_2024_08_060.4104
- 16OPENROUTER_CLAUDE_3_5_HAIKU0.3774
- 17GEMINI_2_0_FLASH0.3726
- 18OPENROUTER_MISTRAL_MEDIUM_30.3538
- 19MISTRAL_CODESTRAL_25010.3349
- 20OPENROUTER_COHERE_COMMAND_A0.3113
- 21GEMINI_2_0_FLASH_LITE0.3066
- 22OPENROUTER_LLAMA_4_MAVERICK0.2736
- 23QWEN_2_5_CODER_32B_INSTRUCT0.2594
- 24OPENROUTER_AMAZON_NOVA_PRO_1_00.25
- 25GPT_4O_MINI0.2311
- 26GROK_2_12120.2217
- 27GROK_3_MINI_BETA_HIGH0.1981
- 28OPENROUTER_LLAMA_4_SCOUT0.184
- 29GPT_4_1_NANO0.1509
- 30LLAMA_3_3_70B_INSTRUCT0.1226
- 31OPENROUTER_QWEN_TURBO0.0802
- 32OPENROUTER_MISTRAL_CODESTRAL_MAMBA0.0142