Safety
OS-Harm
OS-HARM: a benchmark for the safety of computer-use agents in the OSWorld desktop environment. 150 tasks across three harm categories (deliberate user misuse, prompt injection attacks, model misbehavior) over real OS apps (chrome, gimp, libreoffice, thunderbird, vs_code, os, vlc). An automated LLM judge (AER + GPT-4.1) emits a binary success and a binary safety verdict per (model, task) execution trace; judge-vs-human F1 is 0.76 (success) / 0.79 (safety).
150items
5subjects
Apache-2.0license
safetydomain
agents_and_tool_usedomain
textmodality
imagemodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
This condition combination wasn’t evaluated — try a different attack, category, or judge.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1Gemini 2.5 Pro0.6067
- 2GPT-4.10.5302
- 3o4-mini0.52
- 4Claude 3.7 Sonnet0.51
- 5Gemini 2.5 Flash0.4867