Cybersecurity

CTIBench

CTIBench: a benchmark for evaluating LLMs on cyber threat intelligence tasks, including multiple-choice CTI knowledge (CTI-MCQ), CVE-to-CWE root-cause mapping (CTI-RCM), CVSS vulnerability severity prediction (CTI-VSP), and threat-actor attribution (CTI-TAA). Each task provides per-(model, item) predictions for five LLMs graded against gold answers.

5,483items
5subjects
MITlicense
cybersecuritydomain
knowledgedomain
textmodality
item-level responses released
Saturation status: Unknown

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

CTIBench response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Subjects

  1. 1gpt-4-turbo0.6189
  2. 2Gemini-1.50.5834
  3. 3Llama-3-70B0.5673
  4. 4gpt-3.5-turbo0.5092
  5. 5Llama-3-8B0.4559