Stanford UniversityAI Measurement ScienceAIMS — AI Measurement Science

Reasoning

Correlated Errors in LLMs (HELM MMLU)

Correlated Errors in LLMs — HELM MMLU per-(model x question) responses: 71 LLMs answering ~14K MMLU multiple-choice questions, binary correctness.

13,868items
71subjects
MITlicense
knowledgedomain
reasoningdomain
textmodality
item-level responses released
Saturation status: Unknown
Original sourcePaper← All benchmarks

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

Correlated Errors in LLMs (HELM MMLU) response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Subjects

  1. 1anthropic/claude-3-5-sonnet-202410220.8764
  2. 2anthropic/claude-3-5-sonnet-202406200.872
  3. 3google/gemini-1.5-pro-0020.8641
  4. 4anthropic/claude-3-opus-202402290.8504
  5. 5openai/gpt-4o-2024-08-060.8495
  6. 6meta/llama-3.1-405b-instruct-turbo0.849
  7. 7openai/gpt-4o-2024-05-130.8482
  8. 8openai/gpt-4-06130.8401
  9. 9qwen/qwen2.5-72b-instruct-turbo0.8331
  10. 10google/gemini-1.5-pro-0010.8274
  11. 11qwen/qwen2-72b-instruct0.8274
  12. 12openai/gpt-4-turbo-2024-04-090.8194
  13. 13writer/palmyra-x-0040.8176
  14. 14meta/llama-3.2-90b-vision-instruct-turbo0.8109
  15. 15openai/gpt-4-1106-preview0.8101
  16. 16meta/llama-3.1-70b-instruct-turbo0.8091
  17. 17google/gemini-1.5-pro-preview-04090.8086
  18. 18mistralai/mistral-large-24070.8066
  19. 1901-ai/yi-large-preview0.8
  20. 20meta/llama-3-70b0.7867
  21. 21ai21/jamba-1.5-large0.7824
  22. 22google/text-unicorn@0010.7799
  23. 23writer/palmyra-x-v30.7799
  24. 24google/gemini-1.5-flash-0010.7755
  25. 25google/gemini-1.5-flash-preview-05140.7746
  26. 26qwen/qwen1.5-110b-chat0.7744
  27. 27qwen/qwen1.5-72b0.774
  28. 28microsoft/phi-3-medium-4k-instruct0.7738
  29. 29mistralai/mixtral-8x22b0.7734
  30. 30anthropic/claude-3-sonnet-202402290.762
  31. 3101-ai/yi-34b0.7616
  32. 32microsoft/phi-3-small-8k-instruct0.7613
  33. 33openai/gpt-4o-mini-2024-07-180.7577
  34. 34google/gemma-2-27b0.7481
  35. 35google/gemini-1.5-flash-0020.7402
  36. 36qwen/qwen1.5-32b0.7394
Full data on Hugging FaceBack to the gallery