Skip to main content
AIMSAI Measurement Science

Reasoning & Knowledge

SubjECTive-QA

SubjECTive-QA: measuring subjectivity in earnings-call-transcript QA. Models rate each (question, answer) pair on a 3-class scale (0/1/2) for each of six subjective features (Assertive, Cautious, Clear, Optimistic, Relevant, Specific). Scored as classification correctness against human gold labels.

3,300items
5subjects
98%observed
CC-BY-4.0license
financedomain
textmodality
Original sourcePaperBuild script← All benchmarks

Response matrix

Every model, scored item by item.

Each row is an AI model and each column an item, ordered so the strongest models and easiest items gather toward one corner. 5 subjects × 3,300 items, 98% of cells evaluated.

Fit to width. Hover for subject & item; click a cell for details.

SubjECTive-QA response matrix: AI models (rows) against items (columns)
Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect

Sample items

What the questions look like — and how subjects answer.

A spread of items across the difficulty range, each shown with a few subjects' actual answers.

Item 10% solve rateanswer: 1

Given the following feature: OPTIMISTIC and its corresponding definition: The speaker answers with a positive tone regarding outcomes. Give the answer a rating of: 2: If the answer positively demonstrates the chosen feature, with regards to the question. 1: If there is no evident/neutral correlation between the question and the answer for the feature. 0: If the answer negatively correlates to the question on the chosen feature. Provide the rating only. No explanations. This is the question: Yes, thank you. Just a couple of questions. Dr. Paul, do you have any sense of whether or not the book in trends here which obviously are weak to book-to-bill as weak as it’s spending a couple of years, is your sense that bookings are starting to bottom out here, and do you get a sense that revenues could bottom out here or could it take another quarter into the March quarter for this correction to play out? and this is the answer: Well Matt, what we have seen is a kind of a turn around since the beginning of October so orders in absolute numbers come back and get stronger which also means something of course for book-to-bill ration. Really, September, my appreciation was a low point. Future will tell of course, but as we see a continued trend, still relatively weak order rates, but definitely better than they were in September. So which degree this will continue to be? Hard to say. But I think personally, that unless the world economy offers another surprise, the worst may be over..

How subjects answered

  • Llama-3-70b-chat-hf incorrect

    {'id': '88a1850ff8ab9a66-DXB', 'status': 'finished', 'prompt': ['Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: OPTIMISTIC and its corresponding definition:The speaker answers with a positive tone regarding outcomes.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • Llama-3-8b-chat-hf incorrect

    id='8b390759fb733af7-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723723276 model='meta-llama/llama-3-8b-chat-hf' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.Length: 'length'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\ …

  • Mixtral-8x7B-Instruct-v0.1 incorrect

    id='8b3c4637aac74174-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723757306 model='mistralai/mixtral-8x7b-instruct-v0.1' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.EOS: 'eos'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=' 2', tool_calls=None), seed=9817344866000814000)] prompt=[] usage=UsageData(prompt_tokens=367, completion_tokens=3, total_tokens=370)

  • gpt-4o incorrect

    ChatCompletion(id='chatcmpl-9wRvhhOpilELcoqlNWDCf1PHMLEJH', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='2', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1723717413, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_2a322c9ffc', usage=CompletionUsage(completion_tokens=1, prompt_tokens=333, total_tokens=334))

Item 20% solve rateanswer: 1

Given the following feature: OPTIMISTIC and its corresponding definition: The speaker answers with a positive tone regarding outcomes. Give the answer a rating of: 2: If the answer positively demonstrates the chosen feature, with regards to the question. 1: If there is no evident/neutral correlation between the question and the answer for the feature. 0: If the answer negatively correlates to the question on the chosen feature. Provide the rating only. No explanations. This is the question: Yeah. Thanks. And well everyone is stay healthy. I just wanted to follow-up, Jayshree, on your commentary from the beginning of the call you said, you're planning to run the business versus 2018 levels. I assume that's an OpEx comment?Or is it an OpEx comment that also maybe relates to where you think revenue could kind of tends to allow, at the end of the year? and this is the answer: And Ita and I want to reiterate something, we will absolutely continue peddle metal invest in key R&D and customer support. We will reduce marketing travel, obviously, since we can't travel and perhaps some IT spending. And so we work on what we can control and manage the business. And if it improves, we'll certainly recalibrate and invest more..

How subjects answered

  • Llama-3-70b-chat-hf incorrect

    {'id': '88a1ae08892117df-DXB', 'status': 'finished', 'prompt': ["Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: OPTIMISTIC and its corresponding definition:The speaker answers with a positive tone regarding outcomes.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • Llama-3-8b-chat-hf incorrect

    id='8b3921bc8ab63b2e-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723724357 model='meta-llama/llama-3-8b-chat-hf' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.Length: 'length'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2', tool_calls=None), seed=1507142807559088000)] prompt=[] usage=UsageData(prompt_tokens=291, completion_tokens=512, total_tokens=803)

  • Mixtral-8x7B-Instruct-v0.1 incorrect

    id='8b3c5b71beaa3aab-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723758175 model='mistralai/mixtral-8x7b-instruct-v0.1' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.EOS: 'eos'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=' 2', tool_calls=None), seed=6399710138389562000)] prompt=[] usage=UsageData(prompt_tokens=330, completion_tokens=3, total_tokens=333)

  • gpt-4o incorrect

    ChatCompletion(id='chatcmpl-9wS9Sa6HmOAKMzTmGb31rPZ3LKVxZ', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='2', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1723718266, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_2a322c9ffc', usage=CompletionUsage(completion_tokens=1, prompt_tokens=284, total_tokens=285))

Item 30% solve rateanswer: 1

Given the following feature: SPECIFIC and its corresponding definition: The speaker includes specific and technical details in the answer. Give the answer a rating of: 2: If the answer positively demonstrates the chosen feature, with regards to the question. 1: If there is no evident/neutral correlation between the question and the answer for the feature. 0: If the answer negatively correlates to the question on the chosen feature. Provide the rating only. No explanations. This is the question: Just a quick follow-up. So you had more than 20 patients on this current patch formulation in the Phase 1 trials? and this is the answer: The dataset you are seeing is derived from 20 patients. There have been a lot more different variations, a number of different iterations of this patch that have been evaluated over the course of the last 18 months..

How subjects answered

  • Llama-3-70b-chat-hf incorrect

    {'id': '8897274c38bf12be-DXB', 'status': 'finished', 'prompt': ['Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: SPECIFIC and its corresponding definition:The speaker includes specific and technical details in the answer.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • Llama-3-8b-chat-hf incorrect

    id='8b34540dceb9f44a-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723673989 model='meta-llama/llama-3-8b-chat-hf' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.Length: 'length'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2', tool_calls=None), seed=7966585331323181000)] prompt=[] usage=UsageData(prompt_tokens=200, completion_tokens=512, total_tokens=712)

  • Mixtral-8x22B-Instruct-v0.1 incorrect

    {'id': '88bdc0c1cb72b47f-DXB', 'status': 'finished', 'prompt': ['Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: SPECIFIC and its corresponding definition:The speaker includes specific and technical details in the answer.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • Mixtral-8x7B-Instruct-v0.1 incorrect

    id='8b3c33401cc4445b-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723756528 model='mistralai/mixtral-8x7b-instruct-v0.1' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.EOS: 'eos'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=' 0', tool_calls=None), seed=3297523540493505000)] prompt=[] usage=UsageData(prompt_tokens=229, completion_tokens=3, total_tokens=232)

  • gpt-4o incorrect

    ChatCompletion(id='chatcmpl-9wBJgnULQ001k9ABiQbZrbxk2IP7w', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='2', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1723653552, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_2a322c9ffc', usage=CompletionUsage(completion_tokens=1, prompt_tokens=197, total_tokens=198))

Item 420% solve rateanswer: 0

Given the following feature: SPECIFIC and its corresponding definition: The speaker includes specific and technical details in the answer. Give the answer a rating of: 2: If the answer positively demonstrates the chosen feature, with regards to the question. 1: If there is no evident/neutral correlation between the question and the answer for the feature. 0: If the answer negatively correlates to the question on the chosen feature. Provide the rating only. No explanations. This is the question: Okay. So, it’s safe to assume this whole credit crunch that happened, you didn’t see your commercial paper rates sky rocket or anything that was undesirable? and this is the answer: Remember we are pretty highly rated so….

How subjects answered

  • gpt-4o correct

    ChatCompletion(id='chatcmpl-9wBctjjhokyCymvu0rnFORpksxagV', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='0', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1723654743, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_2a322c9ffc', usage=CompletionUsage(completion_tokens=1, prompt_tokens=168, total_tokens=169))

  • Llama-3-70b-chat-hf incorrect

    {'id': '889765e30c911ed6-AMS', 'status': 'finished', 'prompt': ['Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: SPECIFIC and its corresponding definition:The speaker includes specific and technical details in the answer.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • Llama-3-8b-chat-hf incorrect

    id='8b3478a1baf03c7c-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723675486 model='meta-llama/llama-3-8b-chat-hf' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.Length: 'length'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\ …

  • Mixtral-8x22B-Instruct-v0.1 incorrect

    {'id': '88bde7854fd6b478-DXB', 'status': 'finished', 'prompt': ['Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: SPECIFIC and its corresponding definition:The speaker includes specific and technical details in the answer.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • Mixtral-8x7B-Instruct-v0.1 incorrect

    id='8b3c5153ccc885cf-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723757761 model='mistralai/mixtral-8x7b-instruct-v0.1' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.EOS: 'eos'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=' 2: The answer by the speaker indicates a specific and technical detail about their credit rating, which is related to the question about commercial paper rates not skyrocketing during the credit crunch.', tool_calls=None), seed=7388116101034721000)] prompt=[] usage=UsageData(prompt_tokens=200, completion_tokens=40, total_tokens=240)

Item 533% solve rateanswer: 1

Given the following feature: CAUTIOUS and its corresponding definition: The speaker answers using a more conservative, risk-averse approach. Give the answer a rating of: 2: If the answer positively demonstrates the chosen feature, with regards to the question. 1: If there is no evident/neutral correlation between the question and the answer for the feature. 0: If the answer negatively correlates to the question on the chosen feature. Provide the rating only. No explanations. This is the question: Jeff, looking at the 2% to 4% internal revenue growth guidance, what are kind of the one or two biggest swing factors, particularly get to the high end of that range? and this is the answer: Yes, I would say it's the biggest opportunity that exists today. And earlier in 2010, early in the year in '10 -- sorry, I was getting my year straight -- we had talked about the fact that there were some larger transactions that we were in pursuit of. Some of them we could announce publicly, and some of them we could not announce publicly. And one of those happen to be WestPac, and we announced that it was a large transaction. Well, during the year, we ended up winning the transactions we thought we would be able to win. We didn't announce them because they were not announceable, but these are transactions that will be implemented over time. And the constraints that we have are really around having to contend with resource constraints within these very large financial institutions. We have to effectively get into their development queues. And some of the projects that we are working on, to the extent they can be moved in earlier, would have upside into our numbers for 2011. So that's the upside. I think on the downside, to the extent that the economy itself begins to deteriorate down to levels below what we saw last year in 2010, I think that would create some downside in our numbers. Beyond that, I would say that on balance, we would say there's probably more opportunity in our numbers if the economy stays as strong as people are talking about it staying right now. But we just don't see any reason, Chris, to get ahead of ourselves. We feel like we've got a reputation of doing what you say you're going to do, and that's what we're focused on right now. .

How subjects answered

  • Llama-3-70b-chat-hf correct

    {'id': '8898090899c51fb8-HKG', 'status': 'finished', 'prompt': ["Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: CAUTIOUS and its corresponding definition:The speaker answers using a more conservative, risk-averse approach.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • Llama-3-8b-chat-hf incorrect

    id='8b337ab368333c5e-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723665086 model='meta-llama/llama-3-8b-chat-hf' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.Length: 'length'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2', tool_calls=None), seed=7665561610123892000)] prompt=[] …

  • Mixtral-8x22B-Instruct-v0.1 incorrect

    {'id': '88be4ce17c55b47e-DXB', 'status': 'finished', 'prompt': ["Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: CAUTIOUS and its corresponding definition:The speaker answers using a more conservative, risk-averse approach.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • Mixtral-8x7B-Instruct-v0.1 incorrect

    id='8b3bfc67abbb41c7-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723754284 model='mistralai/mixtral-8x7b-instruct-v0.1' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.EOS: 'eos'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=' Based on the given definition of the feature "CAUTIOUS", the provided answer would be rated as a 2. The speaker does provide a more conservative, risk-averse approach when answering the question by acknowledging both potential upsides and downsides, while also expressing caution towards making assumptions about economic conditions and maintaining a focus on delivering …

  • gpt-4o incorrect

    ChatCompletion(id='chatcmpl-9w1sI2BmeaTfo3L3ytHAYYjtPVcbI', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='2', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1723617258, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_2a322c9ffc', usage=CompletionUsage(completion_tokens=1, prompt_tokens=497, total_tokens=498))

Item 640% solve rateanswer: 1

Given the following feature: OPTIMISTIC and its corresponding definition: The speaker answers with a positive tone regarding outcomes. Give the answer a rating of: 2: If the answer positively demonstrates the chosen feature, with regards to the question. 1: If there is no evident/neutral correlation between the question and the answer for the feature. 0: If the answer negatively correlates to the question on the chosen feature. Provide the rating only. No explanations. This is the question: Thanks. Good morning. Just two questions from me if I could. First one on the funnel expansion, you mentioned some decisions by some customers to move to outsourcing. Could you expand on what drove that decision? Whether it was something that was already in the works or maybe if there was a change in thinking due to circumstances, and then I had to follow-up? and this is the answer: Yes. This is Steve. In some of these challenging times, I think customers do question their supply chain strategies more. And so I think the conversations we’re having right now are things that customers were considering, but I think the COVID situation is pushing them to move a little bit quicker. So not a big strategic shift on – I mean, a big strategic shift on their part, but I think the timing of COVID is just helping expedite things..

How subjects answered

  • Llama-3-70b-chat-hf correct

    {'id': '88a12ca2e8ea17df-DXB', 'status': 'finished', 'prompt': ['Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: OPTIMISTIC and its corresponding definition:The speaker answers with a positive tone regarding outcomes.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • gpt-4o correct

    ChatCompletion(id='chatcmpl-9wRQvu2ZGSmmEu2EQWnnKD8DqWoeP', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='1', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1723715505, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_845eaabc1f', usage=CompletionUsage(completion_tokens=1, prompt_tokens=291, total_tokens=292))

  • Llama-3-8b-chat-hf incorrect

    id='8b38cf5589cbf405-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723720982 model='meta-llama/llama-3-8b-chat-hf' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.Length: 'length'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2', tool_calls=None), seed=5080721955010349000)] prompt=[] usage=UsageData(prompt_tokens=295, completion_tokens=512, total_tokens=807)

  • Mixtral-8x22B-Instruct-v0.1 incorrect

    {'id': '88bf063faea91266-DXB', 'status': 'finished', 'prompt': ['Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: OPTIMISTIC and its corresponding definition:The speaker answers with a positive tone regarding outcomes.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • Mixtral-8x7B-Instruct-v0.1 incorrect

    id='8b3c1835fbd985e4-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723755421 model='mistralai/mixtral-8x7b-instruct-v0.1' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.EOS: 'eos'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=' 2', tool_calls=None), seed=8528912617671859000)] prompt=[] usage=UsageData(prompt_tokens=324, completion_tokens=3, total_tokens=327)

Item 760% solve rateanswer: 2

Given the following feature: ASSERTIVE and its corresponding definition: The speaker answers with certainty about the company's events. Give the answer a rating of: 2: If the answer positively demonstrates the chosen feature, with regards to the question. 1: If there is no evident/neutral correlation between the question and the answer for the feature. 0: If the answer negatively correlates to the question on the chosen feature. Provide the rating only. No explanations. This is the question: And then just two other quick ones. For you Chris, and I will just quickly ask them then you can answer. For you Chris, just I want to make sure I understand the free cash flow generation that you are looking for ‘15 and ‘16 and then how much you will be offsetting by VCC spending. And really where I am going with this is I am trying to understand what your usable free cash flow or available free cash flow is for other activities over the next two years and your capital allocation sort of objectives what that is. And then I don't know if I missed this but the whole blood revenues that you are actually expecting for fiscal ‘15 and how much share repurchase is actually incorporated into your guidance. and this is the answer: That’s all right. So in terms of the plan and the guidance we provided, we’ve made an assumption about how much of the share buyback we would get after this year. Of course, we’re going to have pricing grid and we’re going to buy at some prices and not others. So you have to make a somewhat arbitrary assumption about that, for I think conservatively we said $50 million of share buyback. Other than that, our capital allocation policy is pretty clear, acquisitions, service debt, buyback shares. And we’ve told you what we think we will commit to buying back shares and we will get after it at an appropriate rate as market conditions permit..

How subjects answered

  • Llama-3-70b-chat-hf correct

    {'id': '888c3c3b6f4999ed-CDG', 'status': 'finished', 'prompt': ["<s>[INST] <<SYS>> Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier. <</SYS>> Given the following feature: ASSERTIVE and its corresponding definition:The speaker answers with certainty about the company's events.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the …

  • Mixtral-8x22B-Instruct-v0.1 correct

    {'id': '88ae6eceb837b476-DXB', 'status': 'finished', 'prompt': ["Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: ASSERTIVE and its corresponding definition:The speaker answers with certainty about the company's events.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • gpt-4o correct

    ChatCompletion(id='chatcmpl-9w0K7tqNyZxslyXWu2SVK6vMPVOR1', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='2', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1723611295, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_2a322c9ffc', usage=CompletionUsage(completion_tokens=1, prompt_tokens=411, total_tokens=412))

  • Llama-3-8b-chat-hf incorrect

    id='8b39521f7d9f427e-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723726338 model='meta-llama/llama-3-8b-chat-hf' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.Length: 'length'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='1assistant\n\n2assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\ …

  • Mixtral-8x7B-Instruct-v0.1 incorrect

    id='8b3bf8107ac2444f-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723754104 model='mistralai/mixtral-8x7b-instruct-v0.1' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.EOS: 'eos'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=' 1', tool_calls=None), seed=14692566443264457000)] prompt=[] usage=UsageData(prompt_tokens=454, completion_tokens=3, total_tokens=457)

Item 875% solve rateanswer: 1

Given the following feature: OPTIMISTIC and its corresponding definition: The speaker answers with a positive tone regarding outcomes. Give the answer a rating of: 2: If the answer positively demonstrates the chosen feature, with regards to the question. 1: If there is no evident/neutral correlation between the question and the answer for the feature. 0: If the answer negatively correlates to the question on the chosen feature. Provide the rating only. No explanations. This is the question: Well, I was just wondering if you have any or what the contingency plans are, supposing the data does not live up to expectations, supposing you have to increase your trial size, I mean, how does that plan or play out in terms of licensing the product? and this is the answer: I understand. Well, first of all, we will look at the data and understand a lot about it, when Mark has the chance to get the full analysis from the outside vendor. So, there are many different ways that data could come out, certainly if for some unexpected reason, it was clearly negative and there was evidence of a futility here and we would stop the trial as you know GenVec has substantial efforts in other areas. Well funded programs in the areas of vaccine development primarily but also we have other important programs. So, I think we are in a fortunate position compared to many companies in which the technology is broadly ample. So, we are not just focused on TNFerade as it relates to the future but the pipeline, which is driven by the vaccine programs, Doug, actually highlighted couple of important points. One, is the RSV and the HSV-2 program, which we will notice as the company is beginning to build through outside funding, proprietary programs which could also drive the value of the company..

How subjects answered

  • Llama-3-70b-chat-hf correct

    {'id': '88a0ff5a5d7a17db-DXB', 'status': 'finished', 'prompt': ['Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: OPTIMISTIC and its corresponding definition:The speaker answers with a positive tone regarding outcomes.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • Llama-3-8b-chat-hf correct

    id='8b38b4ba0c0f4808-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723719892 model='meta-llama/llama-3-8b-chat-hf' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.Length: 'length'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='1assistant\n\n2assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\n\n1assistant\ …

  • gpt-4o correct

    ChatCompletion(id='chatcmpl-9wRBnYtiUm97H4nbIiabMjvItxDFv', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='1', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1723714567, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_2a322c9ffc', usage=CompletionUsage(completion_tokens=1, prompt_tokens=388, total_tokens=389))

  • Mixtral-8x7B-Instruct-v0.1 incorrect

    id='8b3c025a6d0785ea-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723754526 model='mistralai/mixtral-8x7b-instruct-v0.1' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.EOS: 'eos'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=' 2', tool_calls=None), seed=10222571910262670000)] prompt=[] usage=UsageData(prompt_tokens=423, completion_tokens=3, total_tokens=426)

Item 980% solve rateanswer: 2

Given the following feature: RELEVANT and its corresponding definition: The speaker has answered the question entirely and appropriately. Give the answer a rating of: 2: If the answer positively demonstrates the chosen feature, with regards to the question. 1: If there is no evident/neutral correlation between the question and the answer for the feature. 0: If the answer negatively correlates to the question on the chosen feature. Provide the rating only. No explanations. This is the question: Good morning everyone. Maureen, I had a question on the free cash flow. If you look at where you were in fiscal '15 and what you're guiding to the fiscal '16, you've got a number of puts and takes there, kind of one-time items. What is the more normalized free cash flow level that we should expect for the business going forward, now that Ed Tech is not part of the mix? And then on a related note, where should we expect cash -- your ending cash balance to be at the end of fiscal '16? Thanks. and this is the answer: Okay. So Drew, as far as our ongoing cash flow, the transaction-related fees are 20 million, that's a one-time item, and the 10 million of Ed Tech cash flow contribution, well, that will be out of our numbers. So I think you can expect going forward that our cash flow would be very similar to our historical levels, but 10 million less for Ed Tech..

How subjects answered

  • Llama-3-70b-chat-hf correct

    {'id': '886b119fbdc7f0df-CDG', 'status': 'finished', 'prompt': ["<s>[INST] <<SYS>> Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier. <</SYS>> Given the following feature: RELEVANT and its corresponding definition:The speaker has answered the question entirely and appropriately.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the …

  • Llama-3-8b-chat-hf correct

    id='8b3292226d033b0d-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723655560 model='meta-llama/llama-3-8b-chat-hf' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.Length: 'length'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2', tool_calls=None), seed=10912253339757312000)] prompt=[] usage=UsageData(prompt_tokens=327, completion_tokens=512, total_tokens=839)

  • Mixtral-8x7B-Instruct-v0.1 correct

    id='8b3c4908ae59f37f-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723757421 model='mistralai/mixtral-8x7b-instruct-v0.1' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.EOS: 'eos'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=' 2', tool_calls=None), seed=14490769931925530000)] prompt=[] usage=UsageData(prompt_tokens=365, completion_tokens=3, total_tokens=368)

  • gpt-4o correct

    ChatCompletion(id='chatcmpl-9vsnSCYtLPl9cq9LBVN4HAKp7qt9q', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='2', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1723582362, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_2a322c9ffc', usage=CompletionUsage(completion_tokens=1, prompt_tokens=321, total_tokens=322))

  • Mixtral-8x22B-Instruct-v0.1 incorrect

    {'id': '88ae4648ee4ab47e-DXB', 'status': 'finished', 'prompt': ["Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: RELEVANT and its corresponding definition:The speaker has answered the question entirely and appropriately.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

Item 1080% solve rateanswer: 2

Given the following feature: CLEAR and its corresponding definition: The speaker is transparent in the answer and about the message to be conveyed. Give the answer a rating of: 2: If the answer positively demonstrates the chosen feature, with regards to the question. 1: If there is no evident/neutral correlation between the question and the answer for the feature. 0: If the answer negatively correlates to the question on the chosen feature. Provide the rating only. No explanations. This is the question: And I guess what I'm trying to get at is, is this kind of new way of kind of targeting consumers like the return that you see from what you've done like -- can you just give us any kind of understanding, because one thing I'm noticing across the entire CPG landscape is companies are spending more money, but they're spending more money on the same message and in fact new consumers are being recruited, usage occasions are changing, and so it really requires a change in how you talk to those consumers. And so, I find this Google partnership incredibly compelling for you guys. So, I'm just trying to get better context on how it's changing the return profile of your spend? If you can provide any context on that. and this is the answer: So, we have a continuous focus on optimizing media and the returns on our media. We always have, we've had strong media ROIs forever and our challenge is, how do we keep making them better and then we continuously optimize based on that. So right now, it's a big spend area for us. So it's an area we're very focused on elevating and it is about making sure right now, I would say some of the biggest opportunity is that opportunity of even more precise targeting and then once you have that target, the ability to alter the message. So -- and then we alter our media mix accordingly. And I would say we see significant movement in that mix on a year-to-year basis, as we get better at that. I think we've raised some of the opportunities of how we've gone just very deep relative to specific seasons, whether it's S'mores by zip code or whether it's specific holidays and looking at sell-through at a store level basis to be able to dial-up media on a zip code level basis. So you'll continue to see more of that, frankly that's just becoming a way of how we operate now..

How subjects answered

  • Llama-3-70b-chat-hf correct

    {'id': '8895d6f3bed30272-CDG', 'status': 'finished', 'prompt': ["Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: CLEAR and its corresponding definition:The speaker is transparent in the answer and about the message to be conveyed.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the …

  • Llama-3-8b-chat-hf correct

    id='8b309d9c8b0846c0-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723635059 model='meta-llama/llama-3-8b-chat-hf' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.Length: 'length'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2', tool_calls=None), seed=14639315622618636000)] prompt=[] usage=UsageData(prompt_tokens=518, completion_tokens=512, total_tokens=1030)

  • Mixtral-8x22B-Instruct-v0.1 correct

    {'id': '88a97d8b59069a66-DXB', 'status': 'finished', 'prompt': ["Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: CLEAR and its corresponding definition:The speaker is transparent in the answer and about the message to be conveyed.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the …

  • Mixtral-8x7B-Instruct-v0.1 correct

    id='8b3bcf569f79f399-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723752437 model='mistralai/mixtral-8x7b-instruct-v0.1' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.EOS: 'eos'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=' I would rate the answer as a 2 for clearly demonstrating the feature of "CLEAR." The speaker directly addresses the question about their focus on optimizing media returns and mentions their efforts towards more precise targeting and messaging customization. They also discuss their data-driven approach to adjusting their media mix and highlight specific examples of their …

  • gpt-4o incorrect

    ChatCompletion(id='chatcmpl-9vr2W8Dswrp3tmyfwAoQJZrRppzPb', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='1', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1723575608, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_2a322c9ffc', usage=CompletionUsage(completion_tokens=1, prompt_tokens=499, total_tokens=500))

Item 11100% solve rateanswer: 2

Given the following feature: CLEAR and its corresponding definition: The speaker is transparent in the answer and about the message to be conveyed. Give the answer a rating of: 2: If the answer positively demonstrates the chosen feature, with regards to the question. 1: If there is no evident/neutral correlation between the question and the answer for the feature. 0: If the answer negatively correlates to the question on the chosen feature. Provide the rating only. No explanations. This is the question: Okay. So in terms of the 2018 outlook then, what are your expectations? Do we continue to sort of see a flattish or declines in SG&A? You put out a target of 10% to 10.5% of sales, SG&A and R&D the last quarter around. Do you see downside – could that be lower given the revenue trends? And then just in terms of the ERP investment, how does that sort of play into it? Do we see a higher ERP spend in the first half of this year? Just sort of give us an outlook on sort of SG&A and how that trends. and this is the answer: Yes. Well, we stick our guidance of those numbers that you mentioned in terms of SG&A as a percent of revenue. In terms of the ERP system, as I said before, we’ll have additional costs but at the same time we’re taking cost out. We’re basically going – as I said before, we’re going to the cloud, so we will not – we will gradually get off our AS400 legacy type platforms. But for – I would say mostly in the third and fourth quarter you’ll see an uptick in IT costs but you also will see corresponding – as it relates to the cloud, but also some reductions in other IT costs as well as other SG&A costs..

How subjects answered

  • Llama-3-70b-chat-hf correct

    {'id': '885529ce4a5f697e-CDG', 'status': 'finished', 'prompt': ['<s>[INST] <<SYS>> Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier. <</SYS>> Given the following feature: CLEAR and its corresponding definition:The speaker is transparent in the answer and about the message to be conveyed.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates …

  • Llama-3-8b-chat-hf correct

    id='8b3042473e1c3c88-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723631318 model='meta-llama/llama-3-8b-chat-hf' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.Length: 'length'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2', tool_calls=None), seed=2606659749010671600)] prompt=[] usage=UsageData(prompt_tokens=400, completion_tokens=512, total_tokens=912)

  • Mixtral-8x22B-Instruct-v0.1 correct

    {'id': '88a8f52f2d9f3cda-CDG', 'status': 'finished', 'prompt': ['Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: CLEAR and its corresponding definition:The speaker is transparent in the answer and about the message to be conveyed.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the …

  • Mixtral-8x7B-Instruct-v0.1 correct

    id='8b3b80d6df2c40cf-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723749221 model='mistralai/mixtral-8x7b-instruct-v0.1' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.EOS: 'eos'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=' 2', tool_calls=None), seed=14714108593775903000)] prompt=[] usage=UsageData(prompt_tokens=457, completion_tokens=3, total_tokens=460)

  • gpt-4o correct

    ChatCompletion(id='chatcmpl-9vqGsNGIRYUqhqyH4GgsdmkMLR0v1', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='2', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1723572654, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_2a322c9ffc', usage=CompletionUsage(completion_tokens=1, prompt_tokens=398, total_tokens=399))

Item 12100% solve rateanswer: 2

Given the following feature: ASSERTIVE and its corresponding definition: The speaker answers with certainty about the company's events. Give the answer a rating of: 2: If the answer positively demonstrates the chosen feature, with regards to the question. 1: If there is no evident/neutral correlation between the question and the answer for the feature. 0: If the answer negatively correlates to the question on the chosen feature. Provide the rating only. No explanations. This is the question: OK. Got you. And just to clarify, last time you said that all three segments were expected to grow revenue, adjusted operating income, and adjusted EBITDA. Is that still the case? and this is the answer: It is. In fact, the performance in the first quarter makes us even more confident in our ability to execute a very good year, and the team is performing at a very high level. So yes, we feel very confident about the guidance that we had provided earlier this year..

How subjects answered

  • Llama-3-70b-chat-hf correct

    {'id': '888e426a8aed99a4-CDG', 'status': 'finished', 'prompt': ["Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: ASSERTIVE and its corresponding definition:The speaker answers with certainty about the company's events.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • Llama-3-8b-chat-hf correct

    id='8b39fb95dd433c58-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723733280 model='meta-llama/llama-3-8b-chat-hf' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.Length: 'length'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2assistant\n\n2', tool_calls=None), seed=2824512821424178700)] prompt=[] usage=UsageData(prompt_tokens=227, completion_tokens=512, total_tokens=739)

  • Mixtral-8x22B-Instruct-v0.1 correct

    {'id': '88bce3bdbf07125e-DXB', 'status': 'finished', 'prompt': ["Discard all the previous instructions. Behave like you are an expert sentence clas-\n sifier.\n\n Given the following feature: ASSERTIVE and its corresponding definition:The speaker answers with certainty about the company's events.\n\n Give the answer a rating of:\n\n 2: If the answer positively demonstrates the chosen feature, with regards to the question.\n\n 1: If there is no evident/neutral correlation between the question and the answer for the feature.\n\n 0: If the answer negatively correlates to the question on the chosen …

  • Mixtral-8x7B-Instruct-v0.1 correct

    id='8b3c57188ad2f452-BOM' object=<ObjectType.ChatCompletion: 'chat.completion'> created=1723757997 model='mistralai/mixtral-8x7b-instruct-v0.1' choices=[ChatCompletionChoicesData(index=0, logprobs=None, finish_reason=<FinishReason.EOS: 'eos'>, message=ChatCompletionMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=' 2', tool_calls=None), seed=4706411867082425000)] prompt=[] usage=UsageData(prompt_tokens=253, completion_tokens=3, total_tokens=256)

  • gpt-4o correct

    ChatCompletion(id='chatcmpl-9w1IJdS8iFEGiL3AgGtBt8Jnu5ihK', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='2', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1723615027, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_845eaabc1f', usage=CompletionUsage(completion_tokens=1, prompt_tokens=221, total_tokens=222))

Subjects

The models, agents, and reward models evaluated.

5 subjects, ranked by mean response (accuracy) across this benchmark's items.

  1. 1Mixtral-8x22B-Instruct-v0.10.5879
  2. 2Llama-3-8b-chat-hf0.5642
  3. 3Llama-3-70b-chat-hf0.5571
  4. 4Mixtral-8x7B-Instruct-v0.10.5038
  5. 5gpt-4o0.4924
Full data on Hugging FaceBack to the gallery