Safety
SAD
SAD (Situational Awareness Dataset): does an LLM know it is an LLM and can it reason about its own situation? 16 subtasks across 7 categories (facts, introspection, stages, self-recognition, anti-imitation, influence, id-leverage), mostly multiple-choice, graded per sample as correct (1) / incorrect (0).
191,727items
24subjects
CC-BY-4.0license
safetydomain
knowledgedomain
reasoningdomain
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1o1-preview-2024-09-120.7072
- 2gpt-3.5-turbo0.6667
- 3ft:gpt-4-0613:dcevals-kokotajlo:world-loc:9IqpWXeU:ckpt-step-7880.6
- 4claude-3-5-sonnet-202406200.5702
- 5o1-mini-2024-09-120.546
- 6claude-3-opus-202402290.5344
- 7Llama-3-70b-chat-hf0.5145
- 8gpt-4-0125-preview0.5099
- 9claude-3-sonnet-202402290.5067
- 10gpt-4-06130.499
- 11gpt-4o0.4908
- 12claude-2.10.4664
- 13claude-instant-1.20.466
- 14claude-3-haiku-202403070.4577
- 15gpt-4-base0.4486
- 16replicate/llama-2-70b-chat0.4272
- 17gpt-3.5-turbo-06130.3994
- 18replicate/llama-2-13b-chat0.3935
- 19replicate/llama-2-7b0.3731
- 20replicate/llama-2-70b0.3701
- 21replicate/llama-2-13b0.3669
- 22replicate/llama-2-7b-chat0.3507
- 23davinci-0020.3372
- 24gpt-40