Skip to main content
AIMSAI Measurement Science

Safety

The Elicitation Game

The Elicitation Game evaluates capability-elicitation techniques against "model organisms" (password-locked and circuit-broken LMs with a hidden WMDP-knowledge capability). Released per-item results give, for each model organism under each elicitation technique (many-shot prefilling, GCG adversarial suffixes, anti-refusal fine-tuning), binary correctness on a 370-question WMDP multiple-choice test set.

370items
6subjects
100%observed
unknownlicense
safetydomain
cybersecuritydomain
knowledgedomain
textmodality
Original sourcePaperBuild script← All benchmarks

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

The Elicitation Game response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Subjects

  1. 1Mistral-7B-v0.1 (circuit-broken organism)0.5793
  2. 2gemma-2-9b-it (circuit-broken organism)0.5378
  3. 3Mistral-7B-Instruct-v0.2 (password-locked organism)0.5347
  4. 4Mistral-7B-v0.1 (password-locked organism)0.5342
  5. 5gemma-2-9b-it (password-locked organism)0.5059
  6. 6Mistral-7B-Instruct-v0.2 (circuit-broken organism)0.3574
Full data on Hugging FaceBack to the gallery