Safety
BBQ
BBQ tests whether models pick the correct answer to multiple choice questions about people from different social groups, asking each question once with too little context to answer it and once with the context that settles it.
999items
81subjects
Apache-2.0license
safetydomain
textmodality
item-level responses released
Saturation status: Yes
Response matrix
Responses come from Stanford CRFM's HELM Safety leaderboard (v1.17.0), which re-ran BBQ 0-shot as forced multiple choice at temperature 0 -- not the original evaluation of Parrish et al.
Rasch analysis p = σ(θ − z)
81,000 responses, 80/20 split over cells · 81 subjects · 999 items
AUC train
0.924
AUC test
0.906
Loading session strips…
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect