ML Engineering & Research
MoE-CAP: a benchmark and framework for sparse Mixture-of-Experts LLM serving systems, characterizing the Cost/Accuracy/Performance trade-off with sparsity-aware utilization metrics (S-MBU, S-MFU). This ingestion captures the released lm-eval-harness per-item correctness: each MoE model under a serving framework is graded per GSM8K / MMLU question.
Response matrix
Each row is an AI model and each column an item, ordered so the strongest models and easiest items gather toward one corner. 14 subjects × 7,736 items, 53% of cells evaluated.
Fit to width. Hover for subject & item; click a cell for details.
Scale: 1 = correct · 0 = incorrect
Sample items
A spread of items across the difficulty range, each shown with a few subjects' actual answers.
The sky is blue because
the Sun mainly emits blue light.
the atmosphere absorbs mostly blue light.
molecules scatter red light more effectively than blue light.
molecules scatter blue light more effectively than red light.
How subjects answered
For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | My son is showing anomalous symptoms, so I figured I should take him to the doctor to get a medication prescription. Scenario 2 | I called the cops on my mom for loving me
Wrong, Wrong
Wrong, Not wrong
Not wrong, Wrong
Not wrong, Not wrong
How subjects answered
Consider the following code segment, which uses the variables r, s, and t. r ← 1 s ← 2 t ← 3 r ← s s ← t DISPLAY (r) DISPLAY (s)
What is displayed as a result of running the code segment?
1 1
1 2
2 3
3 2
How subjects answered
The latitude and longitude coordinates of a place are an example of
a formal region.
a functional region.
an absolute location.
a relative location.
How subjects answered
The competitive market for gasoline, a normal good, is currently in a state of equilibrium. Which of the following would most likely increase the price of gasoline?
Household income falls.
Technology used to produce gasoline improves.
The price of subway tickets and other public transportation falls.
The price of crude oil, a raw material for gasoline, rises.
How subjects answered
Freud said that the penis first becomes the major focus of pleasure-seeking at:
birth
about age 1
between the ages of 3 and 6
puberty
How subjects answered
Subjects
14 subjects, ranked by mean response (accuracy) across this benchmark's items.