{"repo":"Privatris/AgentLeak","free":true,"listed":false,"github":"https://github.com/Privatris/AgentLeak","clone":"git clone https://github.com/Privatris/AgentLeak.git","description":"AgentLeak: Open benchmark for privacy leakage in LLM agents — 7 channels, multi-agent, multi-framework.","language":"Python","stars":30,"topics":["agentic-ai","agents","benchmark","crewai","llm","multi-agent","privacy"],"license":null,"category":"ai-agents","readme_excerpt":"AgentLeak Benchmark for privacy leakage in multi-agent LLM systems. This repository accompanies the IEEE Access paper: AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems . Preprint Paper: https://arxiv.org/abs/2602.11510 IEEE ACCESS Paper: https://ieeexplore.ieee.org/document/11569042/ Key Results (5,694 traces across 5 models) Model C1 (Output) C2 (Internal) H1 (Audit Gap) Total Leak ------- ------------- --------------- ---------------- ------------ Claude-3.5-Sonnet 8.2% 53.9% 45.7% 55.2% GPT-4o 17.2% 76.8% 59.6% 77.6% GPT-4o-mini 41.2% 75.3% 34.2% 76.3% Llama-3.3-70B 26.9% 67.8% 41.3% 89.9% Mistral-Large 47.5% 96.2% 48.7% 99.3% Average 28.2% 74.0% 45.9% 79.7% Key Findings - Internal channels leak 2.6× more than external (74.0% vs 28.2%) - Output-only audits miss 45.9% of violations - Claude 3.5 Sonnet paradox : Lowest C1 leakage (8.2%) but 6.6× internal/external ratio—the highest among all models - Finding 7 (Tool Leakage) : Tool inputs (C3) and system logs (C6) exhibit extremely high leakage rates (up to 85% on Claude 3.5), even when the final agent output (C1) is perfectly sanitized. - Pattern C2 C1 holds across all 5 models tested Scope - 1,000 scenarios (healthcare, finance, legal, corporate) - 7 channels: C1 output, C2 inter-agent, C3-C4 tools, C5 memory, C6 logs, C7 artifacts - 32 attack classes, 6 families - SDK: CrewAI, LangChain, AutoGPT, MetaGPT Reproduction Main Benchmark (C1, C2, C5) To reproduce the main results (Output, In","default_branch":null,"files":null,"tree":[],"storefront":"/r/Privatris","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Privatris/AgentLeak/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}