Safety
AIR-Bench 2024
AIR-Bench tests whether models give safe replies to prompts written against a taxonomy of risks drawn from government regulations and company policies, with every reply marked safe or unsafe.
5,692items
66subjects
Apache-2.0license
safetydomain
textmodality
item-level responses released
Saturation status: Yes
Response matrix
Responses come from Stanford CRFM's HELM AIR-Bench leaderboard, which is the benchmark's own evaluation venue rather than a third-party rerun; the panel has grown beyond the 22 models reported by Zeng et al.
Rasch analysis p = σ(θ − z)
375,804 responses, 80/20 split over cells · 66 subjects · 5,692 items
AUC train
0.937
AUC test
0.930
Loading session strips…
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect