Reasoning & Knowledge
BertaQA: trivia multiple-choice QA contrasting knowledge of Basque local culture vs. global culture. Each item is a 3-way (A/B/C) question; a model is scored correct iff it picks the gold candidate. Questions are evaluated 5-shot in English (en) and Basque (eu).
Response matrix
Each row is an AI model and each column an item, ordered so the strongest models and easiest items gather toward one corner. 6 subjects × 9,489 items, 100% of cells evaluated.
Fit to width. Hover for subject & item; click a cell for details.
Scale: 1 = correct · 0 = incorrect
Sample items
A spread of items across the difficulty range, each shown with a few subjects' actual answers.
Question: A: Georgia B: Czech Republic C: Turkey Answer:
How subjects answered
A
Could you please provide the question or context for these options?
A
Galdera: Noiz sortu zen rock&roll-a dantza gisa? A: 1950ean B: 1952an C: 1955ean Erantzuna:
How subjects answered
A
A
A
Galdera: Nortzuek asmatu zuten pasta? A: Txinatarrek B: Frantsesek C: Italianoek Erantzuna:
How subjects answered
A
C
C
Galdera: Zein garaitako aztarnak aurkitu ziren Kortezubiko Santimamiñe kobetan? A: Behe-paleolitoko aztarnak B: Neolitoko aztarnak C: Epipaleolitoko aztarnak Erantzuna:
How subjects answered
C
A
A
Galdera: Euskarak zenbat genero gramatikal dauka? A: Bat ere ez B: Bi C: Hiru Erantzuna:
How subjects answered
A
C
B
Question: As well as being a Basque actor, producer and film director, Eneko Olasagasti is also a teacher, but where? A: Huhezi B: Bilbao Film School C: The Faculty of Letters in Vitoria-Gasteiz Answer:
How subjects answered
A
A
A
Galdera: Zein da "Zisneen lakua" baletaren egilea? A: Berliotz B: Ravel C: Tchaikovsky Erantzuna:
How subjects answered
C
C
B
Galdera: Hiru liburu hauetako zein idatzi zuen Stephen King-ek? A: "Drakula" B: "Gulliver-en bidaiak" C: "Distira" Erantzuna:
How subjects answered
C
C
C
Galdera: Nongoa da Ilintxa futbol-taldea? A: Legazpikoa B: Barakaldokoa C: Galdakaokoa Erantzuna:
How subjects answered
A
A
A
Question: The "Lakalaka dance" is typical of which country? A: Of the Congo B: Of Madagascar C: Of Tonga Answer:
How subjects answered
C
C
C
Question: What is the largest object in the solar system? A: Neptune B: Jupiter C: The sun Answer:
How subjects answered
C
C
C
Galdera: Zer da David artelana? A: Margolana B: Eskultura C: Eliza Erantzuna:
How subjects answered
B
B
B
Subjects
6 subjects, ranked by mean response (accuracy) across this benchmark's items.