Safety
SORRY-Bench
SORRY-Bench human-judgment verdicts: per-(model, instruction, prompt-style) binary human labels in {0=refusal, 1=fulfillment}. 31 models x 450 unsafe base instructions across ~20 linguistic mutation styles. Sparse human-annotated sample over the (model x prompt x style) cube.
450items
31subjects
otherlicense
safetydomain
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1aurora-m-biden-harris-redteamed1
- 2mixtral-8x7b-instruct-v0.11
- 3yi-6b-chat1
- 4qwen1.5-7b-chat1
- 5vicuna-13b-v1.51
- 6qwen1.5-32b-chat1
- 7vicuna-33b-v1.31
- 8qwen1.5-1.8b-chat1
- 9dolphin-2.2.1-mistral-7b0.6098
- 10zephyr-7b-beta0.5957
- 11openchat-3.5-01060.5251
- 12mistral-7b-instruct-v0.20.5066
- 13qwen1.5-4b-chat0.5
- 14llama-2-13b-chat0.5
- 15gemini-pro0.3072
- 16vicuna-7b-v1.50.2937
- 17gemma-7b-it0.2804
- 18gpt-4-0125-preview0.2435
- 19gpt-3.5-turbo-01250.2339
- 20gpt-4-06130.2075
- 21llama-2-70b-chat0.1747
- 22llama-2-7b-chat0.1701
- 23gpt-4-1106-preview0.1697
- 24gpt-3.5-turbo-06130.1419
- 25claude-3-opus-202402290.0729
- 26claude-3-haiku-202403070.0727
- 27claude-3-sonnet-202402290.0699
- 28claude-instant-1.20.0608
- 29gpt-3.5-turbo-11060.0591
- 30claude-2.00.0486
- 31claude-2.10.0406