Mathematics
GHOSTS (Mathematical Capabilities of ChatGPT)
GHOSTS: graduate-level natural-language mathematics prompts from the paper 'Mathematical Capabilities of ChatGPT' (Frieder et al., 2023). Each prompt is posed to a language model and the model's answer is graded 1-5 by a human mathematician (higher = better). Subjects are the two ChatGPT snapshots (2023-01-09, 2023-01-30) on the full GHOSTS set and GPT-4 on the miniGHOSTS subset; items span definition retrieval, MATH-dataset problems, textbook exercises (Munkres, Rudin, Durrett), olympiad problems, proof completion and symbolic integration.
707items
3subjects
CC-BY-NC-4.0license
mathematicsdomain
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
lowhighUnobserved
Scale: {1, 2, 3, 4, 5}
Subjects
- 1GPT-44.1471
- 2ChatGPT (2023-01-30)3.2619
- 3ChatGPT (2023-01-09)3.1705