Skip to main content
AIMSAI Measurement Science

Preference

Arena-Hard-Auto

Arena-Hard-Auto: 5-level GPT-4 judgments of models vs a fixed baseline (GPT-4-0314).

500items
72subjects
100%observed
Apache-2.0license
preferencedomain
textmodality
Original sourcePaperBuild script← All benchmarks

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

Arena-Hard-Auto response matrix: AI models (rows) against items (columns)
lowhighUnobserved

Scale: {0, 0.125, 0.25, ..., 1} (8-level judge scale)

Subjects

  1. 1o1-mini-2024-09-120.7907
  2. 2o1-preview-2024-09-120.7752
  3. 3athene-v2-chat0.716
  4. 4llama-3.1-nemotron-70b-instruct0.715
  5. 5claude-3-5-sonnet-202410220.714
  6. 6yi-lightning0.6917
  7. 7gpt-4-turbo-2024-04-090.6847
  8. 8gpt-4o-2024-05-130.6683
  9. 9qwen2.5-72b-instruct0.6677
  10. 10claude-3-5-sonnet-202406200.6657
  11. 11athene-70b-07250.6647
  12. 12gpt-4o-2024-08-060.6548
  13. 13gpt-4-0125-preview0.6538
  14. 14gpt-4o-mini-2024-07-180.6398
  15. 15gemini-1.5-pro-api-05140.6278
  16. 16yi-large-preview0.6245
  17. 17mistral-large-24070.6158
  18. 18llama-3.1-405b-instruct-fp80.608
  19. 19llama-3.1-405b-instruct-fp8-no-sys-prompt0.5847
  20. 20glm-4-05200.5843
  21. 21deepseek-coder-v20.5797
  22. 22yi-large0.5761
  23. 23claude-3-opus-202402290.5587
  24. 24gemma-2-27b-it0.5533
  25. 25glm-4-01160.542
  26. 26llama-3.1-70b-instruct0.5397
  27. 27gemini-1.5-pro-api-0409-preview0.5268
  28. 28glm-4-air0.5175
  29. 29gemini-1.5-flash-api-05140.5075
  30. 30claude-3-sonnet-202402290.4913
  31. 31llama-3-70b-instruct0.4908
  32. 32qwen2-72b-instruct0.4908
  33. 33llama-3.1-nemotron-51b-instruct0.4865
  34. 34gemma-2-9b-it0.468
  35. 35claude-3-haiku-202403070.4597
  36. 36mistral-large-24020.4368
Full data on Hugging FaceBack to the gallery