Skip to main content
AIMSAI Measurement Science

Software Engineering

TDD-Bench-Verified

TDD-Bench-Verified: generating tests from GitHub issues before the fix exists. 449 issues filtered from SWE-bench Verified; a generated test suite passes iff it fails on the pre-fix code and passes on the golden-patched code (fail-to-pass). Per-instance harness verdicts for Otter and e-Otter with GPT-4o and Claude-3.7-Sonnet under ten prompting variants, from the authors' Zenodo artifact.

449items
4subjects
100%observed
apache-2.0license
software_engineeringdomain
textmodality
Original sourcePaperBuild script← All benchmarks

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

TDD-Bench-Verified response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Subjects

  1. 1e-Otter (Claude-3.7-Sonnet)0.3882
  2. 2e-Otter (GPT-4o)0.31
  3. 3Otter (Claude-3.7-Sonnet)0.3086
  4. 4Otter (GPT-4o)0.2465
Full data on Hugging FaceBack to the gallery