Preference
Preference Dissection
Preference Dissection records which of two candidate replies to the same prompt each model judge prefers, turning the judges themselves into the systems being measured.
4,890items
32subjects
CC-BY-NC-4.0license
preferencedomain
textmodality
item-level responses released
Saturation status: No
Response matrix
Subjects are the ~32 AI judges; human annotations are filtered out.
Rasch analysis p = σ(θ − z)
167,680 responses, 80/20 split over cells · 32 subjects · 4,890 items
AUC train
0.939
AUC test
0.925
Loading session strips…
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect