Skip to main content

Reasoning & Knowledge

WikiContradict

WikiContradict: real-world knowledge-conflict QA mined from Wikipedia. Each item is a question whose answer is contradicted across two Wikipedia passages; LLM answers are graded by humans as Correct / Partially correct / Incorrect under several prompt (RAG) templates.

48items
5subjects
100%observed
Apache-2.0license
generaldomain
knowledgedomain
textmodality

Response matrix

Fit to width. Hover for subject & item; click a cell for details.

WikiContradict response matrix: AI models (rows) against items (columns)
lowhighUnobserved

Scale: {0, 0.5, 1} (Incorrect / Partially correct / Correct)

Sample items

Item 128% solve rateanswer: 1965 | 1952

What year did Paul Fraisse become a director of the Institute of Psychology of the University of Paris?

How subjects answered

  • mixtral-8x7b-instruct-v0-1score 0

    Answer: Paul Fraisse became the director of the Institute of Psychology of the University of Paris in 1961.

  • llama-3-70b-instructscore 0

    According to the context, Paul Fraisse became the director of the Institute of Psychology of the University of Paris in 1961, not 1965. In 1965, he was already the director, having taken over the sole directorship in 1961.

  • mistral-7b-instruct-v0-2score 0

    Answer: Paul Fraisse became the sole director of the Institute of Psychology of the University of Paris in 1961, but he was previously part of a director board with Daniel Lagache, Jean Delay, and Pierre-Paul Grassé since 1952. However, Lagache stepped back from co-directorship in 1960, and Fraisse took over as the sole director in 1961. In 1965, Fraisse is mentioned as the director of the Institute, indicating that he had been in that position for several years prior to that year.

  • gpt-4score 0

    Paul Fraisse became the director of the Institute of Psychology of the University of Paris in 1961.

  • llama-2-70b-chatscore 0

    Answer: 1961. According to the context, Paul Fraisse became the only director of the Institute of Psychology of the University of Paris in 1961, after Daniel Lagache stepped back from co-directorship in 1960. However, it's also mentioned that he became the director in 1965, which could suggest that he was appointed as the director in that year, or that he took on additional responsibilities as director in that year. Without further information, it's difficult to determine the exact date of his appointment as director.

Item 242% solve rateanswer: four | three

How many small towns does the valley in which the Mitta Mitta river flows contain?

How subjects answered

  • mistral-7b-instruct-v0-2score 0.5

    Answer: The valley in which the Mitta Mitta river flows contains three small towns: Mitta Mitta, Eskdale, and Dartmouth.

    Explanation: The context states that there are four small towns in the valley, but upon closer inspection, it is clear that one of the towns mentioned, "Mitta Mitta," is actually the name of the river and not a separate town. Therefore, the valley contains three small towns: Eskdale, Dartmouth, and Mitta Mitta (the river).

  • llama-2-70b-chatscore 0.5

    Answer:

    The valley contains 4 small towns.

  • mixtral-8x7b-instruct-v0-1score 0

    Answer: The valley in which the Mitta Mitta river flows contains three small towns. Although the context states that there are four small towns, the answer is three because the context also states that the four towns are Mitta Mitta and Eskdale, which are mentioned twice.

  • llama-3-70b-instructscore 0

    According to the context, the valley in which the Mitta Mitta river flows contains three small towns: Mitta Mitta, Eskdale, and Dartmouth.

  • gpt-4score 0

    The valley in which the Mitta Mitta River flows contains three small towns: Mitta Mitta, Eskdale, and Dartmouth.

Item 348% solve rateanswer: Yes | No

Did the formation of the Belogradchik Rocks start during the Permian period?

How subjects answered

  • mixtral-8x7b-instruct-v0-1score 0.5

    Answer: Yes, the formation of the Belogradchik Rocks did start during the Permian period, around 230 million years ago, as a result of the tectonic cycle that was folding to create the Balkan Mountains and uplifting the Belogradchik region.

  • llama-3-70b-instructscore 0.5

    Yes, the formation of the Belogradchik Rocks started during the Permian period, about 230 million years ago.

  • mistral-7b-instruct-v0-2score 0.5

    However, some geologists argue that the Belogradchik Rocks are actually much older than the Permian period, with some estimates suggesting that they could be as old as 500 million years.

    Answer: The Belogradchik Rocks began to take shape during the Permian period around 230 million years ago, but there is ongoing debate among geologists regarding their true age, with some suggesting they could be up to 500 million years old.

  • gpt-4score 0.5

    Yes, the formation of the Belogradchik Rocks started during the Permian period.

  • llama-2-70b-chatscore 0.5

    Answer: Yes.

Item 458% solve rateanswer: 1746 | 8 January 1747

When the HMS Royal George was laid down?

How subjects answered

  • mixtral-8x7b-instruct-v0-1score 0

    Answer: HMS Royal George was laid down at Woolwich Dockyard on the 8th of January 1747. The war ending in 1748 caused construction to slow down, but the ship was already laid down in 1747.

  • llama-3-70b-instructscore 0

    According to the context, there are two conflicting dates for when the HMS Royal George was laid down: 1746 and January 8, 1747.

  • mistral-7b-instruct-v0-2score 0

    Answer: HMS Royal George was laid down on the 8th of January 1747 at Woolwich Dockyard, but construction was delayed due to the end of the war in 1748.

  • gpt-4score 0

    HMS Royal George was laid down in 1746.

  • llama-2-70b-chatscore 0

    Answer:

    1746 1747 1748

Item 560% solve rateanswer: 1 February 1972 | 10 February 1972

When was Paul McCole born?

How subjects answered

  • mixtral-8x7b-instruct-v0-1score 0

    Answer: Paul McCole was born on 1 February 1972.

  • llama-3-70b-instructscore 0

    According to the provided context, Paul McCole was born on two different dates: 1 February 1972 and 10 February 1972.

  • mistral-7b-instruct-v0-2score 0

    Answer: Paul McCole was born on 1 February 1972 (age 52) according to some sources, while other sources state that he was born on 10 February 1972. Therefore, the exact birthdate of Paul McCole is uncertain and may vary depending on the source.

  • gpt-4score 0

    Paul McCole was born on 1 February 1972.

  • llama-2-70b-chatscore 0

    Answer: Paul McCole was born on February 1, 1972, and February 10, 1972.

Item 662% solve rateanswer: 8 December 1967 | 12 August 1967

When was Marina Baker born?

How subjects answered

  • gpt-4score 0.5

    Marina Baker was born on 8 December 1967.

  • mixtral-8x7b-instruct-v0-1score 0

    Marina Baker is a fictional character.

    Answer: The Marina Baker mentioned in the context was born on 8 December 1967, but it is important to note that this Marina Baker is a fictional character.

  • llama-3-70b-instructscore 0

    According to the provided context, Marina Baker's birthdate is contradictory, with two different dates mentioned: 8 December 1967 and 12 August 1967.

  • mistral-7b-instruct-v0-2score 0

    Marina Baker is the daughter of Marina Augusta Baker.

    Answer: Marina Baker was born on 8 December 1967. She is the daughter of Marina Augusta Baker, who was born on 12 August 1961 (mistake in the context, should be 1967), making Marina Baker's birth year consistent with the given information.

  • llama-2-70b-chatscore 0

    Marina Baker was born on 12 August 1967.

    Answer: Marina Baker was born on December 8, 1967, and on August 12, 1967.

Item 764% solve rateanswer: 1919 | 1918

When did Pierre Albert-Birot's novel Grabinoulor first appear?

How subjects answered

  • mixtral-8x7b-instruct-v0-1score 0.5

    Answer: Pierre Albert-Birot's novel Grabinoulor first appeared in 1919, but the dinners named after the novel and its eponymous character, Grabinoulor, began in 1936.

  • llama-3-70b-instructscore 0.5

    According to the context, Pierre Albert-Birot's novel Grabinoulor first appeared in 1919, but it was also mentioned that the novel was started in 1918. Therefore, the answer is that Grabinoulor first appeared in 1919, but its creation began in 1918.

  • mistral-7b-instruct-v0-2score 0

    Answer: Pierre Albert-Birot's novel Grabinoulor was first published in 1919, but it gained renewed significance and became the namesake for regular gatherings of Albert-Birot's old friends starting in 1936.

  • gpt-4score 0

    Pierre Albert-Birot's novel Grabinoulor first appeared in 1919.

  • llama-2-70b-chatscore 0

    Answer: 1918, 1919, 1936.

Item 864% solve rateanswer: 20 acres (81,000 m2) | 20.5 acres (83,000 m2)

How large is the Tregaron estate in both acres and square meters?

How subjects answered

  • gpt-4score 0.5

    The Tregaron estate covers 20.5 acres or 83,000 square meters.

  • mixtral-8x7b-instruct-v0-1score 0

    Answer: The Tregaron estate is 20.5 acres (83,000 m2) in size, which is equivalent to approximately 83,000 square meters. However, after the Washington International School and the Tregaron Limited Partnership made their purchases, the estate was divided into two parts: 6 acres (24,000 m2) and 14 acres (57,000 m2).

  • llama-3-70b-instructscore 0

    According to the context, the Tregaron estate is approximately 20-20.5 acres in size, which is equivalent to around 81,000-83,000 square meters.

  • mistral-7b-instruct-v0-2score 0

    Answer: The Tregaron estate covers a total area of 83,000 m2 (20.5 acres). After Davies's death, 6 acres (24,000 m2) were sold to the Washington International School, and the remaining 14 acres (57,000 m2) were purchased by an Israeli corporation. Therefore, the Washington International School now owns approximately 27% of the estate, and the Israeli corporation owns approximately 65%. The exact size of each parcel in square meters is not provided in the context.

  • llama-2-70b-chatscore 0

    Answer: The Tregaron estate is 20.5 acres or 83,000 square meters.

Item 966% solve rateanswer: 5-metre (16 ft) | 2 meters (6 ft 7 in)

What is the maximum firing range of the Blaster (flamethrower)?

How subjects answered

  • mixtral-8x7b-instruct-v0-1score 0

    Answer: The maximum firing range of the Blaster (flamethrower) is 5 meters (16 ft) as stated in the first context, but it is also mentioned in the second context that the maximum firing range is 2 meters (6 ft 7 in) flame. Both statements are provided in the context, and they seem to contradict each other. However, it is essential to acknowledge both pieces of information to reflect the comprehensive view of the context.

  • llama-3-70b-instructscore 0

    According to the context, there is a discrepancy in the maximum firing range of the Blaster (flamethrower). Initially, it is stated that the flamethrower can direct 5-meter (16 ft) plumes of flame, but later it is mentioned that the maximum firing range is 2 meters (6 ft 7 in) flame. Therefore, the maximum firing range of the Blaster is unclear, with two different ranges mentioned: 5 meters (16 ft) and 2 meters (6 ft 7 in).

  • mistral-7b-instruct-v0-2score 0

    Answer: The Blaster flamethrower has a maximum firing range of 2 meters (6 ft 7 in) according to the provided context, despite the earlier statement suggesting a 5-meter (16 ft) range when used against carjackers attempting to enter a vehicle. This discrepancy may be due to the context's focus on the upward direction of the flame when used defensively against attackers, potentially increasing the perceived range. However, the actual maximum firing range is 2 meters (6 ft 7 in).

  • gpt-4score 0

    The maximum firing range of the Blaster (flamethrower) is 2 meters (6 ft 7 in).

  • llama-2-70b-chatscore 0

    Answer: The maximum firing range of the Blaster (flamethrower) is 5 meters (16 ft) and 2 meters (6 ft 7 in) flame, depending on the context.

Item 1068% solve rateanswer: It is held together by overlapping segments joined by wood glue. | It is held together by wooden pegs, with no glue.

How is the Loretto Chapel staircase held together?

How subjects answered

  • mistral-7b-instruct-v0-2score 1

    Answer: The Loretto Chapel staircase is held together by its twisted helical stringers, which function similarly to a newel or central pole, and by wooden pegs that prevent degradation of the joints due to changes in humidity or temperature. Despite popular belief, the staircase does not rely solely on wooden pegs and does not lack any glue or nails. Instead, the stringers are assembled from overlapping segments joined by wood glue, creating a stronger laminate than the wood alone. The use of wooden pegs instead of nails further enhances the staircase's durability.

  • mixtral-8x7b-instruct-v0-1score 0.5

    Answer: The Loretto Chapel staircase is held together by its stringers, which are twisted into a helix and joined by wooden pegs. The inner stringer has a tight radius that functions similarly to a straight center support, and the assembly of the stringers from overlapping segments creates a laminate that is stronger than the wood alone. The use of wooden pegs rather than nails prevents degradation of the joints due to compression set as the wood swells against the nails due to changes in humidity or temperature.

  • llama-3-70b-instructscore 0.5

    The Loretto Chapel staircase is held together by a combination of twisted stringers, wooden pegs, and a unique laminate structure created by overlapping segments of wood joined with wood glue. The tight radius of the inside stringer also provides additional support, similar to a straight center pole. Despite the lack of a newel or central pole, the staircase's innovative design and construction materials allow it to support its weight and remain stable.

  • gpt-4score 0.5

    The Loretto Chapel staircase is held together primarily by wooden pegs, with the stringers constructed from overlapping segments of a unique type of spruce wood. These segments are joined tightly, allowing the inner stringer to function similarly to a central support due to its tight radius. Despite the absence of nails, glue, or other hardware, the staircase maintains its structural integrity through the careful assembly of these wood segments and the natural properties of the wood used.

  • llama-2-70b-chatscore 0.5

    Answer: The Loretto Chapel staircase is held together by wooden pegs, without the use of glue, nails, or other hardware. The staircase's stringers, which are twisted into a helix, provide support for the weight of the staircase, with the inside stringer functioning similarly to a straight center support. The assembly of the stringers from overlapping segments joined by wood glue creates a laminate that is stronger than the wood alone.

Item 1172% solve rateanswer: August 21st 1968 | 27 August 1968

What date was Julian Lewis Jones born?

How subjects answered

  • llama-3-70b-instructscore 0.5

    According to the provided context, Julian Lewis Jones was born on either 21 August 1968 or 27 August 1968, as there are two conflicting birthdates mentioned.

  • gpt-4score 0.5

    Julian Lewis Jones was born on 21 August 1968.

  • mixtral-8x7b-instruct-v0-1score 0

    Julian Lewis Jones was born on 21 August 1968 in the Welsh town of Cowbridge. He is a Welsh actor and producer, best known for his role as Mr. Bridger in the 2015 film 'Kingsman: The Secret Service'. Jones has also appeared in numerous television shows, including 'EastEnders' and 'Stella'. In addition to his acting career, Jones is also a successful producer and has worked on several films and television shows. He currently resides in Cardiff, Wales with his wife and two children.

  • mistral-7b-instruct-v0-2score 0

    Julian Lewis Jones was born on the 21st of August, 1968. Julian Lewis Jones was born on the 27th of August, 1968.

    Answer: Julian Lewis Jones was born on the 21st of August, 1968, according to some sources, while others state that he was born on the 27th of August, 1968. The exact date of his birth remains a matter of conflicting information.

  • llama-2-70b-chatscore 0

    Julian Lewis Jones, born 21st August 1968 Julian Lewis Jones, born on August 27, 1968) is a Welsh actor.

    Answer:

    The correct answer is August 21, 1968, and August 27, 1968.

    Explanation: The context provides two different dates of birth for Julian Lewis Jones. The first mention states that he was born on August 21, 1968, while the second and third mentions state that he was born on August 27, 1968. It is possible that the discrepancy in dates is a typo or an error in reporting. Therefore, both dates should be considered as the answer.

Item 1276% solve rateanswer: Yes | No

Is the Baltic Sea considered a type of mediterranean sea?

How subjects answered

  • mixtral-8x7b-instruct-v0-1score 0.5

    Answer: Yes, the Baltic Sea is considered a type of mediterranean sea, as it is an example of a mediterranean sea that is a dilution basin, with a lower salinity due to freshwater gains and water exchange consisting of outflow of fresher mediterranean water and inflow of saltier oceanic water. However, it is also considered not to be a mediterranean sea because it is a brackish inland sea, alleged to be the largest body of brackish water in the world, occupying a basin formed by glacial erosion.

  • llama-3-70b-instructscore 0.5

    The answer is both yes and no. The Baltic Sea can be considered a type of mediterranean sea in the sense that it is a dilution basin with a specific water exchange pattern, as described in the context. However, it is also argued that the Baltic Sea is not a mediterranean sea because it is a brackish inland sea, distinct from the typical characteristics of a mediterranean sea. This contradictory information reflects the heterogeneous nature of the context.

  • mistral-7b-instruct-v0-2score 0.5

    Answer: The Baltic Sea is considered a brackish inland sea and not a typical Mediterranean sea, despite being referred to as a Mediterranean sea of the Atlantic Ocean in some contexts. Its unique hydrological features, such as being a dilution basin with a lower salinity and limited deep water renewal, distinguish it from the Mediterranean seas characterized by high salinity and strong evaporation.

  • gpt-4score 0.5

    Yes, the Baltic Sea is considered a type of mediterranean sea, specifically a dilution basin with unique characteristics such as lower salinity and a large body of brackish water, despite some definitions suggesting it is merely a brackish inland sea.

  • llama-2-70b-chatscore 0.5

    Answer: No, the Baltic Sea is not considered a mediterranean sea. Although it is sometimes referred to as a mediterranean sea due to its location and connection to the Atlantic Ocean, it is actually a brackish inland sea with a unique hydrological system that sets it apart from true mediterranean seas.

Subjects

  1. 1llama-3-70b-instruct0.6542
  2. 2mixtral-8x7b-instruct-v0-10.6229
  3. 3gpt-40.6083
  4. 4llama-2-70b-chat0.5917
  5. 5mistral-7b-instruct-v0-20.5813