{"repo":"nottelabs/open-operator-evals","free":true,"listed":false,"github":"https://github.com/nottelabs/open-operator-evals","clone":"git clone https://github.com/nottelabs/open-operator-evals.git","description":"Opensource benchmark evaluating web operators/agents performance","language":"Python","stars":47,"topics":["ai-agents","ai-tools","browser-automation","browser-use","computer-use","cua","llm","notte","web-agent"],"license":null,"category":"ai-agents","readme_excerpt":"Opensource operators evals Rank Provider Agent Self-Report LLM Evaluation Time per Task Task Reliability ---- ----------------------------------------------------------- ----------------- -------------- ------------- ---------------- 🏆 Notte 86.2% 79.0% 47s 96.6% 2️⃣ Browser-Use 77.3% 60.2% 113s 83.3% 3️⃣ Convergence 38.4% 31.4% 83s 50% Key highlights You can investigate all replays/logs and reproduce the benchmark yourself 👇🏻 - Notte leads the benchmark by achieving the highest performance with 86.2% self-reported success and 79% LLM-verified completion. It also has the fastest execution time at 47s per task and an impressive 96.6% task reliability—Percentage of tasks an agent successfully completes at least once when given multiple attempts - Our evaluation of Browser-Use shows 77.3% self-reported agent performance and 60.2% LLM-verified success, compared to their reported 89% in their blog post. Their results files are not available for verification. - Convergence achieved 38.4% agent success and 31.4% evaluation success, lower than other methods. Performance was limited by CAPTCHA and bot detection. Notably, it showed strong self-awareness, with near-perfect alignment in some cases, suggesting potential if detection issues are addressed. PS: We are actively hiring software and research engineers 🪩 The metrics In the main table - Agent Self-Report The success rate reported by the agent itself across all tasks. This reflects the agent's internal confidence in its perfor","default_branch":null,"files":null,"tree":[],"storefront":"/r/nottelabs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/nottelabs/open-operator-evals/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}