{"repo":"SAP/agent-quality-inspect","free":true,"listed":false,"github":"https://github.com/SAP/agent-quality-inspect","clone":"git clone https://github.com/SAP/agent-quality-inspect.git","description":"Evaluation package that allows benchmarking of agentic AIs from various sources and frameworks by producing statistical results which can be compared across different use cases and datasets.","language":"Python","stars":79,"topics":["agentic-ai","error-analysis","evaluation","llm","llm-as-a-judge","metrics","user-proxy"],"license":"Apache-2.0","category":"ai-agents","readme_excerpt":"Agent Quality Inspect Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis (ICLR 2026) Paper Link: https://openreview.net/pdf?id=fHsVNklKOc Documentation Link: https://sap.github.io/agent-quality-inspect/ Table of Contents - Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis - Table of Contents - Overview - Features - Installation - Prerequisites - Setup - Quick Start - Option 1. Using it as a Metrics Package - Option 2. Using it via the provided runners - Viewing Results - Agents Leaderboard UI - Bring Your Own Agent - Creating your own evaluation dataset - Known Issues - How to obtain support - Contributing - Citation - License Overview This repository contains the implementation of Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis (TED) . The agent-quality-inspect toolkit evaluates agentic systems under different user personas (expert and non-expert), reports metrics such as Area Under the Curve (AUC) , Progress Per Turn (PPT) , pass@k , pass^k , etc., and provides detailed error analysis to identify specific areas for improvement in the agent. At the core of TED is a subgoal-based evaluation : users specify a set of natural-language subgoals (e.g., \"Agent should call search messages after getting the current timestamp\" or \"Agent should explicitly state the final answer and justification\") in the evaluation dataset. During evaluation, TED: - Treats these subgoals (the SubGoal ","default_branch":null,"files":null,"tree":[],"storefront":"/r/SAP","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/SAP/agent-quality-inspect/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}