Cybersecurity
NYU CTF Bench
NYU CTF Bench: 200 Capture-The-Flag challenges from CSAW competitions (crypto, forensics, pwn, rev, web, misc) for evaluating LLMs and agents in offensive security. A subject solves a challenge if it recovers the challenge flag.
200items
9subjects
MITlicense
cybersecuritydomain
software_engineeringdomain
agents_and_tool_usedomain
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.

Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1claude-3.5-sonnet-202410220.2088
- 2claude-3.7-sonnet-202502190.1805
- 3claude-3.5-sonnet-202406200.135
- 4gpt-4.10.1263
- 5gpt-4o0.1068
- 6gpt-4-1106-preview0.07
- 7gpt-4-0125-preview0.05
- 8gpt-3.5-turbo-11060.04
- 9claude-3-haiku-202403070.04