Multilingual
BertaQA
BertaQA: trivia multiple-choice QA contrasting knowledge of Basque local culture vs. global culture. Each item is a 3-way (A/B/C) question; a model is scored correct iff it picks the gold candidate. Questions are evaluated 5-shot in English (en) and Basque (eu).
9,489items
6subjects
CC-BY-SA-4.0license
culturaldomain
multilingualdomain
knowledgedomain
textmodality
item-level responses released
Saturation status: Unknown
Response matrix
Fit to width. Hover for subject & item; click a cell for details.
Correct (1)Incorrect (0)Unobserved
Scale: 1 = correct · 0 = incorrect
Subjects
- 1claude-3-opus-202402290.8155
- 2gpt-4-0125-preview0.8069
- 3gpt-4-06130.7761
- 4claude-3-sonnet-202402290.7111
- 5claude-3-haiku-202403070.703
- 6gpt-3.5-turbo-01250.628