Multilingual
LINGOLY
LINGOLY: olympiad-level linguistic reasoning puzzles in 90+ low-resource and extinct languages, drawn from Linguistics Olympiad problem sheets. A model reads a full problem sheet and answers labelled sub-parts; each part is graded by normalized exact match against the gold answer.
1,133items
11subjects
CC-BY-4.0license
reasoningdomain
multilingualdomain
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1claude-3-opus-202402290.3257
- 2gpt-4o0.259
- 3gpt-4-turbo0.2348
- 4gemini-1.5-pro0.2132
- 5command-r-plus0.1668
- 6gpt-3.5-turbo-01250.1664
- 7Mixtral-8x7B-Instruct-v0.10.12
- 8Meta-Llama-3-8B-Instruct0.1002
- 9Meta-Llama-3-70B-Instruct0.0984
- 10Llama-2-70b-chat-hf0.0697
- 11gemma-7b-it0.049