Skip to main content

Science & Engineering

DataDecide

DataDecide: OLMES multiple-choice QA evaluation of a controlled pretraining suite (25 data recipes x 14 sizes x 3 seeds). Per-instance correctness over 10 OLMES benchmarks (ARC, BoolQ, CommonsenseQA, HellaSwag, MMLU, OpenBookQA, PIQA, Social IQa, Winogrande).

37,553items
1subjects
100%observed
odc-bylicense
generaldomain
reasoningdomain
sciencedomain
textmodality

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

DataDecide response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Sample items

Item 10% solve rate

Question: Jesse paid another bill for the others so they wouldn't get kicked out. What will Others want to do next? Answer:

  • A.

    pay jesse back later

  • B.

    yell at jesse

  • C.

    find out how much the bill was

Subject outcomes

  • DataDecide OLMo dclm-baseline 750M seed-14 step-25000 incorrect
Item 20% solve rate

Question: A 54-year-old woman comes to the emergency department because of severe jaw pain since undergoing a painful dental procedure 1 day ago. The patient was prescribed codeine after the procedure and instructed to take the medication every 4 hours, but she has continued pain despite adherence to this analgesic regimen. Other members of her family also have experienced poor pain control with codeine. Which of the following is the most likely explanation for this therapeutic failure? Answer:

  • A.

    Decreased absorption of codeine

  • B.

    Decreased metabolism of codeine to morphine

  • C.

    Deficiency of κ receptors

  • D.

    Increased plasma protein-binding of codeine

Subject outcomes

  • DataDecide OLMo dclm-baseline 750M seed-14 step-25000 incorrect
Item 30% solve rate

Alcohol laws of Australia -- Alcohol laws of Australia are laws that regulate the sale and consumption of alcoholic beverages. The legal drinking age is 18 throughout Australia. The minimum age for the purchase of alcoholic products in Australia is 18. A licence to both produce and sell alcohol is required. Question: can you drink alcohol at 16 in australia? Answer:

  • A.

    yes

  • B.

    no

Subject outcomes

  • DataDecide OLMo dclm-baseline 750M seed-14 step-25000 incorrect
Item 40% solve rate

Question: When someone responds to your argument with a sarcastic statement such as, "Yeah, right. Like that's ever going to happen," that person may have committed which fallacy? Answer:

  • A.

    appeal to indignation

  • B.

    appeal to spite

  • C.

    horse laugh

  • D.

    tu quoque

Subject outcomes

  • DataDecide OLMo dclm-baseline 750M seed-14 step-25000 incorrect
Item 50% solve rate

Question: ___________________ began to show up few years back on wireless access points as a new way of adding or connecting new devices. Answer:

  • A.

    WPA2

  • B.

    WPA

  • C.

    WPS

  • D.

    WEP

Subject outcomes

  • DataDecide OLMo dclm-baseline 750M seed-14 step-25000 incorrect
Item 60% solve rate

Goal: how do you walk? Answer:

  • A.

    move around on your feet.

  • B.

    move around on your hands and knees.

Subject outcomes

  • DataDecide OLMo dclm-baseline 750M seed-14 step-25000 incorrect
Item 70% solve rate

Question: What is 'coring'? Answer:

  • A.

    A symptom of phlebitis.

  • B.

    Titration of a drug with its desired effect.

  • C.

    The delivery of medication straight into the bloodstream.

  • D.

    When drugs are withdrawn from a sealed vial, fragments of the rubber bung can be cut out by the needle.

Subject outcomes

  • DataDecide OLMo dclm-baseline 750M seed-14 step-25000 incorrect
Item 8100% solve rate

Question: You are training a linear regression model for a simple estimation task, and notice that the model is overfitting to the data. You decide to add in 2\ell_2 regularization to penalize the weights. As you increase the 2\ell_2 regularization coefficient, what will happen to the bias and variance of the model? Answer:

  • A.

    Bias increase ; Variance increase

  • B.

    Bias increase ; Variance decrease

  • C.

    Bias decrease ; Variance increase

  • D.

    Bias decrease ; Variance decrease

Subject outcomes

  • DataDecide OLMo dclm-baseline 750M seed-14 step-25000 correct
Item 9100% solve rate

Relationships: How to make him want you back. Accept the break-up. Handle the break-up maturely and with grace. If he broke up with you, respect his decision and give him some time to see things more clearly.

  • A.

    Remember that this is your break-up, not his, as he may feel rejected or like there's no room for you in his life. Don't assume that is all there is to a break-up and change things up if it becomes difficult.

  • B.

    Just because you broke up with him doesn't mean you have to push him away. He may still be interested in you but put a lot of effort into rebuilding his relationship so that you can either pursue things you normally wouldn't have done-have fun together, go on dates, or get intimate.

  • C.

    He will appreciate your understanding and respect you all the more for handling yourself with dignity. The last thing you want to do is make a scene.

  • D.

    Give him some time to think this over before you continue the conversation. Feeling angry, hurt, or betrayed is never good for a relationship.

Subject outcomes

  • DataDecide OLMo dclm-baseline 750M seed-14 step-25000 correct
Item 10100% solve rate

Pets and Animals: How to keep a dog in good health. Feed your dog high-quality, well-balanced dog food and treats. This should be your pet's main source of nutrition. Look at the first five ingredients listed on the pet food label.

  • A.

    These ingredients make up the majority of the food. Meat (not meat by-products) and vegetables should be the first few ingredients in the dog food.

  • B.

    If the first ingredient appears " fat " on the ingredient list, then the food is high quality. A high quality dog food is high in protein because it is low in fat.

  • C.

    Typically, these will be listed below the food. If the dog food label lists one or two ingredients, just put a little bit of the first ingredient into the second ingredient on your to-do list, then continue on with the food.

  • D.

    You don't want a high-quality food over low quality. Low quality foods encourage choking problems and make it harder for your dog to handle the treats.

Subject outcomes

  • DataDecide OLMo dclm-baseline 750M seed-14 step-25000 correct
Item 11100% solve rate

Question: Which question about tulips could best be answered by scientific research? Answer:

  • A.

    Are tulips better than other flowers?

  • B.

    What genes determine tulip petal color?

  • C.

    Why do people like to look at tulips?

  • D.

    Which color of tulips is the prettiest?

Subject outcomes

  • DataDecide OLMo dclm-baseline 750M seed-14 step-25000 correct
Item 12100% solve rate

Question: Why are Cepheid stars relevant for astronomers? Answer:

  • A.

    To measure interstellar mass.

  • B.

    To measure galactic distances.

  • C.

    To measure galactic energy-density.

  • D.

    To measure interstellar density.

Subject outcomes

  • DataDecide OLMo dclm-baseline 750M seed-14 step-25000 correct

Subjects

  1. 1DataDecide OLMo dclm-baseline 750M seed-14 step-250000.4468