Software Engineering

SWE-rebench

SWE-rebench: an automated, decontaminated benchmark of real-world software-engineering agent tasks. Each task is a GitHub issue + repo snapshot; an LLM agent must produce a patch verified by the repo test suite (FAIL_TO_PASS / PASS_TO_PASS). This build ingests the publicly released per-instance trajectories of the OpenHands v0.54.0 + Qwen3-Coder-480B-A35B-Instruct agent, emitting a binary resolved=1/0 response per (agent, instance, run).