Safety

CASE-Bench

CASE-Bench evaluates whether LLM safety judgments align with human judgments when the context of a harmful-looking query is taken into account. 900 items pair 450 SORRY-Bench queries with a safe and an unsafe Contextual-Integrity-formalized context; each item carries a human majority label from ~21 crowd annotators. Released per-item outputs cover 7 LLM judges under three elicitation methods (direct binary judgment, 1-10 score thresholded at >5.5, token-probability thresholded at >0.5); the response is 1 if the model judgment matches the human majority label.