{"repo":"fdabench/FDAbench","free":true,"listed":false,"github":"https://github.com/fdabench/FDAbench","clone":"git clone https://github.com/fdabench/FDAbench.git","description":"🔥[KDD’26] Official repository for the paper “FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data”.","language":"Python","stars":80,"topics":["agents","ai","chatgpt","data","data-analysis","data-science","data-visualization","database","dataset","llama"],"license":"MIT","category":"databases-storage","readme_excerpt":"FDABench (KDD'26) FDABench is a benchmark for evaluating data agents on analytical queries over heterogeneous data. It contains 2,007 tasks across diverse domains, difficulty levels, and task types. We provide ready-to-use data agent implementations, a DAG-based evaluation system, and an agent-expert collaboration framework for dataset generation. Overview Key Features - Data Agent Implementations : Ready-to-use agents (Planning, Multi-Agent, Reflection, Tool-Use) - Comprehensive Evaluation : Accuracy metrics, rubric-based scoring, and performance analytics - Multi-Database Support : SQLite, BigQuery, Snowflake, and unstructured data sources - Three Task Types : Single-choice, multiple-choice, and open-ended report generation - Extensible Framework : Modular base classes for custom agent integration - Cost Tracking : Token usage and latency monitoring Task Types Type Description ------ ------------- Single Choice One correct answer from four options Multiple Choice Multiple correct answers allowed Report Open-ended analytical report generation Environment Setup System Requirements - Python: 3.10+ - OS: Linux, macOS, Windows Option 1: One-Command Setup Create the complete environment with all dependencies: This will: - Create a new conda environment named fdabench - Install Python 3.11 and all required dependencies - Automatically install FDABench in development mode Option 2: Manual Setup If you prefer manual installation: API Configuration Set up your API keys for LLM access","default_branch":null,"files":null,"tree":[],"storefront":"/r/fdabench","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/fdabench/FDAbench/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}