Skip to main content
AIMSAI Measurement Science

Preference

JudgeTuning

JudgeTuning: LLM-as-judge design-decision tuning. Per-battle human agreement of tuned open-weight LLM judges (Ours-tiny/small/medium/large) and reference judges (Arena-Hard GPT-4o-mini, JudgeLM-7B, PandaLM-7B) on 3,000 LMSys pairwise preference battles.

2,981items
7subjects
100%observed
apache-2.0license
reward_modelingdomain
preferencedomain
textmodality
Original sourcePaperBuild script← All benchmarks

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

JudgeTuning response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Subjects

  1. 1Arena-Hard judge (GPT-4o-mini)0.5005
  2. 2JudgeTuning Ours-large (tuned open-weight LLM judge <72B)0.4935
  3. 3JudgeTuning Ours-medium (tuned open-weight LLM judge <32B)0.4717
  4. 4JudgeTuning Ours-small (tuned open-weight LLM judge <10B)0.4572
  5. 5JudgeTuning Ours-tiny (tuned open-weight LLM judge)0.4418
  6. 6JudgeLM-7B0.4257
  7. 7PandaLM-7B0.3784
Full data on Hugging FaceBack to the gallery