Agents & Tool Use
REAL: autonomous web agents on deterministic simulations of real websites. Tasks across high-fidelity website clones (Staynb, DashDish, ...), each with a deterministic success checker made of one or more evals. Response is the per-attempt binary task success (1 iff all of the task's evals pass).
Response matrix
Each row is an AI model and each column an item, ordered so the strongest models and easiest items gather toward one corner. 44 subjects × 233 items, 42% of cells evaluated.
Fit to width. Hover for subject & item; click a cell for details.
Scale: 1 = correct · 0 = incorrect
Sample items
A spread of items across the difficulty range. This benchmark does not publish per-answer traces, so each item shows which subjects succeeded.
Show me places in Provence, France for 2 adults and 1 child for the dates August 1st-4th. I need wifi.
Subject outcomes
Compare all the stays listed by review quality and price and rank them by affordability. Please provide a ranked list showing each stay's name, rating, and price.
Subject outcomes
Create a job post for a UI/UX Designer with expertise in Figma, Sketch, and Adobe Creative Suite, including project details, timeline, and required skills (Wireframing, Prototyping, Responsive Design).
Subject outcomes
Find and book the stay with the best value for money (cheapest stay with the best reviews) for 1 day. For fields you don't know the answer for, just fill them in with anything of your choice.
Subject outcomes
Find a home in San Diego priced under $150,000 with at least 2 bedrooms and request a tour. Use these details: Contact Name: John Doe, Email: johndoe@example.com, Phone: 555-123-4567, Tour Time: 2:00 PM, Tour Date: First available.
Subject outcomes
Filter the listings in San Jose to display only "Townhomes" within a price range of 1,000,000. How many results are displayed?
Subject outcomes
Mark the first email in the Inbox as "read".
Subject outcomes
Show me places in San Francisco with wifi for the dates September 27-29th for 2 people.
Subject outcomes
Email Brian King and tell him to send me the meeting notes.
Subject outcomes
List the top 5 most recently shared posts on the homepage.
Subject outcomes
Book me a ride from 1001 Castro Street to 1030 Post Street Apartments.
Subject outcomes
DashDish: Place an order for any type of sub-sandwich, keep the total under $30
Subject outcomes
Subjects
44 subjects, ranked by mean response (accuracy) across this benchmark's items.