{"repo":"lechmazur/confabulations","free":true,"listed":false,"github":"https://github.com/lechmazur/confabulations","clone":"git clone https://github.com/lechmazur/confabulations.git","description":"Hallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.","language":"HTML","stars":249,"topics":["benchmark","claude","gemini","hallucinations","leaderboard","llama","llm","rag","confabulations","ai-evaluation"],"license":null,"category":"ai-agents","readme_excerpt":"LLM Confabulation (Hallucination) Leaderboard for RAG This benchmark evaluates large language models (LLMs) based on how frequently they produce non-existent answers (confabulations or hallucinations) in response to misleading questions that are based on provided text documents. These documents are recent articles not yet included in the LLM training data. Minimizing these types of confabulation is crucial when using Retrieval-Augmented Generation (RAG). The questions are intentionally crafted to be challenging. As of Feb 10, 2025, 201 questions, confirmed by a human to lack answers in the provided texts, have been carefully curated and assessed. The number of questions will ideally increase over time, and the error margins will continue to narrow. However, the current dataset is already sufficient to differentiate between various models. --- Combined Evaluation Combining confabulation and non-response rates enables a comprehensive ranking. Depending on your priorities, you may prefer fewer confabulations or fewer non-responses. 50-50 Ranking bar chart 50-50 Ranking leaderboard Rank Model Confab % Non-Resp % Weighted ---: --- ---: ---: ---: 1 GPT-5 (medium reasoning) 10.9 9.8 10.34 2 Gemini 2.5 Pro Preview 05-06 5.9 15.3 10.62 3 Grok 3 Mini Beta (high) 6.9 14.7 10.80 4 Gemini 2.5 Pro Exp 03-25 4.0 17.6 10.80 5 GLM-4.5 7.9 14.7 11.30 6 o1 (medium reasoning) 10.9 12.6 11.74 7 Qwen 3 30B A3B 12.9 11.7 12.28 8 Gemini 2.5 Pro 4.0 20.8 12.38 9 Grok 4 4.0 20.9 12.41 10 Gemini 2.0 Fl","default_branch":null,"files":null,"tree":[],"storefront":"/r/lechmazur","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/lechmazur/confabulations/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}