{"repo":"OpenAgentHQ/openagent-eval","free":true,"listed":false,"github":"https://github.com/OpenAgentHQ/openagent-eval","clone":"git clone https://github.com/OpenAgentHQ/openagent-eval.git","description":"Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.","language":"Python","stars":15,"topics":["rag","rag-evaluation","cli","llm","metrics","ai","ai-agents","ai-evaluation","benchmark","evaluation"],"license":"Apache-2.0","category":"ai-agents","readme_excerpt":"The open-source evaluation framework for RAG systems and AI Agents. Local-first. Framework-agnostic. Developer-friendly. Getting Started · Documentation · Contributing --- Why OpenAgent Eval? Evaluating RAG systems shouldn't require a PhD or a cloud account. OpenAgent Eval brings pytest-level simplicity to AI evaluation — run from your terminal, get actionable insights, and ship with confidence. - Local-first — No cloud services, dashboards, or authentication required - Framework-agnostic — Works with LangChain, LlamaIndex, or any custom RAG pipeline - 18+ metrics — Retrieval, generation, faithfulness, relevancy, performance, and cost - Plugin-based — Extend with custom metrics, providers, and report generators - Production-ready — Corpus auditing, failure diagnosis, and synthetic test data generation --- Installation For development: --- Quick Start 1. Initialize Configuration The wizard walks you through your dataset path, LLM provider and model, retriever, metric preset, and report format, then writes a ready-to-run config.yaml . 2. Validate Configuration 3. Run Evaluation 4. View Results --- Contributing & Community OpenAgent Eval is community-built and welcomes contributors of every experience level. If you've used the tool, you already know something worth sharing — a bug, a rough edge in the docs, or a metric you wish existed. Ways to help: - Code — pick up a good first issue or something labeled help wanted, fix a bug, or add a metric or provider. - Docs — clarify a g","default_branch":null,"files":null,"tree":[],"storefront":"/r/OpenAgentHQ","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/OpenAgentHQ/openagent-eval/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}