{"repo":"Rogo-Technologies/big-finance-benchmark","free":true,"listed":false,"github":"https://github.com/Rogo-Technologies/big-finance-benchmark","clone":"git clone https://github.com/Rogo-Technologies/big-finance-benchmark.git","description":"Reference harness for the Big Finance benchmark of workflow-grounded financial-research questions","language":"Python","stars":15,"topics":["agents","ai-agents","benchmark","evaluation","finance","llm-evaluation","react","sec-edgar"],"license":"Apache-2.0","category":"ai-agents","readme_excerpt":"Big Finance Harness Reference scaffold for evaluating LLM agents on the Big Finance benchmark — 928 workflow-grounded financial-research questions, each paired with an expert-authored rubric and a reference answer. This harness reproduces the headline numbers from the companion paper, BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents. It is deliberately minimal: a ReAct loop, four publicly-replicable tools, and a unified message format that runs the same scaffold across any model accessible through LiteLLM. Maintained by Rogo Technologies. Contact: open a GitHub issue or email alexwang@rogo.ai . What's here --- --- big finance harness/ Python package: ReAct agent, tools, judge, types scripts/ Orchestrator (eval + grade), analysis, plotting tests/ Test suite (51 tests, no network deps) data/ Public 50-item subset ( big finance subset.jsonl ) + datasheet grades/ Public grading outputs from Gemini 3.1 Pro, Claude Opus 4.7, and GPT-5.5 human workpapers/ Two illustrative workbooks from independent human validation Tools The four tools given to the agent (plus a terminal final answer ): Tool Backed by --- --- web search SerpAPI (preferred) or Tavily (fallback) edgar search SEC EDGAR public REST API fetch url httpx + BeautifulSoup + BM25 (optional in-document retrieval) + PyMuPDF (PDFs) python exec subprocess (5s timeout; not a security sandbox) final answer terminator We deliberately exclude: vector-store retrieval, premium financial data sources (FactSet","default_branch":null,"files":null,"tree":[],"storefront":"/r/Rogo-Technologies","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Rogo-Technologies/big-finance-benchmark/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}