Cybersecurity

NYU CTF Bench

NYU CTF Bench: 200 Capture-The-Flag challenges from CSAW competitions (crypto, forensics, pwn, rev, web, misc) for evaluating LLMs and agents in offensive security. A subject solves a challenge if it recovers the challenge flag.

200items
9subjects
MITlicense
cybersecuritydomain
software_engineeringdomain
agents_and_tool_usedomain
textmodality
item-level responses released
Saturation status: Unknown

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

NYU CTF Bench response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Subjects

  1. 1claude-3.5-sonnet-202410220.2088
  2. 2claude-3.7-sonnet-202502190.1805
  3. 3claude-3.5-sonnet-202406200.135
  4. 4gpt-4.10.1263
  5. 5gpt-4o0.1068
  6. 6gpt-4-1106-preview0.07
  7. 7gpt-4-0125-preview0.05
  8. 8gpt-3.5-turbo-11060.04
  9. 9claude-3-haiku-202403070.04