Stanford UniversityAI Measurement ScienceAIMS — AI Measurement Science

General

Fantastic Bugs

Fantastic Bugs: a measurement-theoretic framework that flags potentially invalid benchmark questions (ambiguous wording, wrong answer key, grading issues) for expert review. Released as per-(model, question) HELM correctness matrices across roughly a dozen widely used benchmarks (GSM8K, MMLU subsets, MedQA, ThaiExam, BBQ, BoolQ, WikiFact, CommonSenseQA, LegalBench, AIR-Bench).

21,905items
180subjects
CC-BY-4.0license
generaldomain
textmodality
item-level responses released
Saturation status: Unknown
Original sourcePaper← All benchmarks

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

Fantastic Bugs response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Subjects

  1. 1anthropic/claude-3-5-sonnet-202410220.8292
  2. 2anthropic/claude-3-5-sonnet-202406200.8131
  3. 3google/gemini-1.5-pro-001-safety-block-none0.7892
  4. 4qwen/qwen2.5-72b-instruct-turbo0.7862
  5. 5anthropic/claude-3-opus-202402290.786
  6. 6amazon/nova-pro-v1:00.7858
  7. 7google/gemini-1.5-pro-preview-04090.7806
  8. 8openai/o1-2024-12-170.772
  9. 9google/gemini-1.5-flash-001-safety-block-none0.7688
  10. 10meta/llama-3.3-70b-instruct-turbo0.7601
  11. 11meta/llama-3-70b0.7588
  12. 12google/gemini-1.5-pro-0010.7563
  13. 13meta/llama-3.2-90b-vision-instruct-turbo0.749
  14. 14google/gemini-2.0-flash-exp0.7466
  15. 15amazon/nova-lite-v1:00.7346
  16. 16writer/palmyra-x-v30.7307
  17. 17upstage/solar-pro-2411260.7259
  18. 18anthropic/claude-3-sonnet-202402290.7246
  19. 19openai/o3-mini-2025-01-310.7216
  20. 20ai21/jamba-1.5-large0.7201
  21. 21google/gemini-1.5-pro-0020.7187
  22. 22mistralai/mixtral-8x22b0.7167
  23. 23openai/gpt-4-turbo-2024-04-090.7164
  24. 24google/gemini-1.5-flash-preview-05140.7156
  25. 25qwen/qwen1.5-72b0.712
  26. 26microsoft/phi-3-medium-4k-instruct0.7096
  27. 27google/gemma-2-27b-it0.7091
  28. 28google/text-unicorn@0010.7078
  29. 29openai/gpt-4-1106-preview0.6968
  30. 30google/gemma-2-27b0.6964
  31. 31openai/gpt-4-06130.6957
  32. 32anthropic/claude-3-5-haiku-202410220.6929
  33. 33google/gemini-1.5-flash-0010.691
  34. 34openai/gpt-4o-2024-08-060.6889
  35. 35qwen/qwen1.5-32b0.6882
  36. 36qwen/qwen2.5-7b-instruct-turbo0.6872
Full data on Hugging FaceBack to the gallery