Skip to main content
AIMSAI Measurement Science

Mathematics

Large Language Monkeys: Power Laws

How Do Large Language Monkeys Get Their Power (Laws)? Per-sample correctness from repeated sampling (10,000 samples per problem) for Llama-3 / Gemma / Pythia models on GSM8K, MATH, MiniF2F-MATH and CodeContests. Each sample is a binary trial (correct / incorrect).

525items
13subjects
42%observed
mitlicense
mathematicsdomain
software_engineeringdomain
reasoningdomain
textmodality
Original sourcePaperBuild script← All benchmarks

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

Large Language Monkeys: Power Laws response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Subjects

  1. 1Llama-3-70B-Instruct0.4142
  2. 2Llama-3-8B-Instruct0.308
  3. 3Gemma-7B0.1008
  4. 4Llama-3-8B0.0748
  5. 5Gemma-2B0.0446
  6. 6Pythia-12B0.0232
  7. 7Pythia-2.8B0.0203
  8. 8Pythia-6.9B0.0186
  9. 9Pythia-1.4B0.0126
  10. 10Pythia-1B0.0083
  11. 11Pythia-410M0.0062
  12. 12Pythia-160M0.0027
  13. 13Pythia-70M0.0006
Full data on Hugging FaceBack to the gallery