{"repo":"lechmazur/deception","free":true,"listed":false,"github":"https://github.com/lechmazur/deception","clone":"git clone https://github.com/lechmazur/deception.git","description":"Benchmark evaluating LLMs on their ability to create and resist disinformation. Includes comprehensive testing across major models (Claude, GPT-4, Gemini, Llama, etc.) with standardized evaluation metrics.","language":null,"stars":33,"topics":["ai-benchmarks","ai-evaluation","ai-safety","ai-security","claude","disinformation","gemini","gpt4o","language-model","llama"],"license":null,"category":"ai-agents","readme_excerpt":"LLM Deceptiveness and Gullibility Benchmark This benchmark assesses large language models along two critical dimensions: their capability to generate convincing disinformation and their resilience against misleading information. The evaluation framework uses recent articles outside the models' training data, deriving fact-based questions that probe both deceptive capabilities and resistance to manipulation. Models must craft persuasive but misleading arguments, while also demonstrating their ability to maintain accurate reasoning when faced with deceptive content from other models. 📊 Methodology Data Collection - Source material consists of recent articles beyond model training cutoffs - Questions are derived from factual content within these articles. - Multiple models evaluate identical questions under varying conditions Evaluation Process 1. Initial Screening : - Approximately 19,000 questions were generated by four models - Models answer questions without exposure to misleading information - Analysis includes only questions where all models achieve perfect scores - This establishes a baseline of correct understanding 3. Deception Phase : - Models create misleading arguments for incorrect answers - Arguments range from direct misinformation to subtle doubt casting - Assessment focuses on argumentative quality and persuasiveness 4. Testing Phase : - Models answer questions while considering misleading arguments - Analysis tracks accuracy and deception resistance - Cross-mo","default_branch":null,"files":null,"tree":[],"storefront":"/r/lechmazur","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/lechmazur/deception/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}