Stanford UniversityAI Measurement ScienceAIMS — AI Measurement Science

General

EgoSchema

EgoSchema: a diagnostic long-form egocentric video question-answering benchmark derived from Ego4D, with 5-way multiple-choice questions about ~3-minute first-person videos. This build ingests real per-(model, item) predictions over the public 500-question label subset from the released LLoVi runs (CeeZh/LLoVi output.zip): 11 LLoVi configurations (LLM backbone + caption source + prompt strategy), each evaluated on the same 500 items. Response is per-item correctness (model choice == gold).

500items
11subjects
mitlicense
reasoningdomain
videomodality
textmodality
item-level responses released
Saturation status: Unknown
Original sourcePaper← All benchmarks

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

EgoSchema response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Subjects

  1. 1LLoVi (GPT-4-1106, standard QA)0.612
  2. 2LLoVi (GPT-3.5-turbo, few-shot + summary)0.602
  3. 3LLoVi (GPT-4, standard QA)0.59
  4. 4LLoVi (GPT-3.5-turbo-1106, QA + summary)0.588
  5. 5LLoVi (GPT-3.5-turbo, few-shot)0.576
  6. 6LLoVi (GPT-3.5-turbo, QA + summary)0.574
  7. 7LLoVi (LLaMA backbone)0.554
  8. 8LLoVi (GPT-3.5-turbo, chain-of-thought)0.552
  9. 9LLoVi (GPT-3.5-turbo-1106, standard QA)0.552
  10. 10LLoVi (GPT-3.5-turbo, standard QA)0.512
  11. 11LLoVi (GPT-3.5-turbo, BLIP-2 captions, standard QA)0.474
Full data on Hugging FaceBack to the gallery