Agents & Tool Use

REAL

REAL tests whether web agents can complete user tasks on deterministic clones of real websites, counting an attempt as successful only when every one of the task's automatic checks passes.

233items

33subjects

unknownlicense

agents_and_tool_usedomain

textmodality

gui_screenshotmodality

item-level responses released

Saturation status: No

Full data on Hugging Face Back to the gallery

Paper and credit

2025

REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (opens in a new tab)

Original source

Authors

Divyansh Garg, Shaun VanWeelden, Diego Caples, Andis Draguns +14 moreShow all

01Divyansh Garg
02Shaun VanWeelden
03Diego Caples
04Andis Draguns
05Nikil Ravi
06Pranav Putta
07Naman Garg
08Tomas Abraham
09Michael Lara
10Federico Lopez
11James Liu
12Atharva Gundawar
13Prannay Hebbar
14Youngchul Joo
15Jindong Gu
16Charles London
17Christian Schroeder de Witt
18Sumeet Motwani

Citation

BibTeX citationShow

BibTeX

1@misc{garg2025realbenchmarkingautonomousagents,
2  title={REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites},
3  author={Divyansh Garg and Shaun VanWeelden and Diego Caples and Andis Draguns and Nikil Ravi and Pranav Putta and Naman Garg and Tomas Abraham and Michael Lara and Federico Lopez and James Liu and Atharva Gundawar and Prannay Hebbar and Youngchul Joo and Jindong Gu and Charles London and Christian Schroeder de Witt and Sumeet Motwani},
4  year={2025},
5  eprint={2504.11543},
6  archivePrefix={arXiv},
7  primaryClass={cs.AI},
8  url={https://arxiv.org/abs/2504.11543},
9}

Response matrix

Loading response matrix…

Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect