General
PertEval
PertEval measures the real knowledge capacity of LLMs by applying knowledge-invariant perturbations to MMLU questions (rephrasing that preserves the required knowledge). Each MMLU item is recast as an option-judgement task: the model labels each of 4 options True/False; an item is correct iff the full True/False vector matches the gold.
4,885items
6subjects
Apache-2.0license
knowledgedomain
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1gpt-4-turbo0.6953
- 2gemini-1.0-pro0.4848
- 3gpt-3.5-turbo0.4328
- 4Meta-Llama-3-8B-Instruct0.421
- 5glm-3-turbo0.387
- 6Mistral-7B-Instruct-v0.20.2768