Safety
HELM AIRBench 2024
HELM AIRBench 2024: per-(model, prompt) air_score binarized to {0,1} (1=safe, threshold 0.5). 66 models x 5694 prompts over the AIR 2024 risk taxonomy (314 categories across 4 levels).
5,692items
66subjects
Apache-2.0license
safetydomain
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1anthropic/claude-3-5-sonnet-202410220.9305
- 2anthropic/claude-sonnet-4-202505140.9009
- 3anthropic/claude-opus-4-202505140.8795
- 4anthropic/claude-3-5-sonnet-202406200.8783
- 5anthropic/claude-3-sonnet-202402290.869
- 6openai/o3-2025-04-160.8681
- 7google/gemini-1.5-pro-001-safety-block-none0.8674
- 8anthropic/claude-3-opus-202402290.8653
- 9anthropic/claude-3-haiku-202403070.8497
- 10anthropic/claude-3-7-sonnet-202502190.8423
- 11openai/o1-2024-12-170.8274
- 12google/gemini-1.5-flash-001-safety-block-none0.8195
- 13writer/palmyra-x50.8163
- 14openai/o4-mini-2025-04-160.8126
- 15openai/o3-mini-2025-01-310.7763
- 16openai/gpt-4.5-preview-2025-02-270.7698
- 17google/gemini-2.5-pro-preview-03-250.7615
- 18openai/gpt-4-turbo-2024-04-090.7543
- 19meta/llama-3-8b-chat0.7355
- 20meta/llama-4-maverick-17b-128e-instruct-fp80.7287
- 21google/gemini-2.0-pro-exp-02-050.7153
- 22google/gemini-2.5-flash-preview-04-170.712
- 23google/gemini-2.0-flash-lite-preview-02-050.7071
- 24google/gemini-1.5-flash-0020.7023
- 25writer/palmyra-fin0.7011
- 26google/gemini-1.5-pro-0020.7006
- 27google/gemini-2.0-flash-0010.6967
- 28openai/gpt-3.5-turbo-03010.6877
- 29openai/gpt-4-06130.6826
- 30openai/gpt-4.1-2025-04-140.6788
- 31openai/gpt-3.5-turbo-06130.6756
- 32meta/llama-3-70b-chat0.6744
- 33openai/gpt-4o-2024-08-060.6621
- 34qwen/qwen2-72b-instruct0.6561
- 35meta/llama-3.1-8b-instruct-turbo0.651
- 36openai/gpt-4.1-nano-2025-04-140.6479