{"repo":"vectara/open-rag-eval","free":true,"listed":false,"github":"https://github.com/vectara/open-rag-eval","clone":"git clone https://github.com/vectara/open-rag-eval.git","description":"RAG evaluation without the need for \"golden answers\"","language":"Python","stars":398,"topics":["evaluation-metrics","rag","retrieval-augmented-generation","vectara","metrics","rag-evaluation"],"license":"Apache-2.0","category":"ai-agents","readme_excerpt":"Open RAG Eval Evaluate and improve your Retrieval-Augmented Generation (RAG) pipelines with open-rag-eval , an open-source Python evaluation toolkit. Evaluating RAG quality can be complex. open-rag-eval provides a flexible and extensible framework to measure the performance of your RAG system, helping you identify areas for improvement. Its modular design allows easy integration of custom metrics and connectors for various RAG implementations. The core metrics (UMBRELA, AutoNuggetizer) do not require golden chunks or golden answers, making RAG evaluation easy and scalable. This is achieved by utilizing UMBRELA and AutoNuggetizer, techniques originating and researched in Jimmy Lin's lab at UWaterloo. Additionally, the toolkit supports optional golden answer evaluation using appropriate metrics when reference answers are available. Out-of-the-box, the toolkit includes: - An implementation of the evaluation metrics used in the TREC-RAG benchmark . - A connector for the Vectara RAG platform . - Connectors for LlamaIndex and LangChain (more coming soon...) Key Features - Standard Metrics: Provides TREC-RAG evaluation metrics ready to use. See METRICS.md for detailed documentation. - Modular Architecture: Easily add custom evaluation metrics or integrate with any RAG pipeline. - Detailed Reporting: Generates per-query scores and intermediate outputs for debugging and analysis. - Visualization: Compare results across different configurations or runs with plotting utilities. Getting ","default_branch":null,"files":null,"tree":[],"storefront":"/r/vectara","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/vectara/open-rag-eval/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}