{"repo":"megagonlabs/holobench","free":true,"listed":false,"github":"https://github.com/megagonlabs/holobench","clone":"git clone https://github.com/megagonlabs/holobench.git","description":"🫧 Code for Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data (Maekawa*, Iso* et al.; ICLR 2025)","language":"Python","stars":12,"topics":["database","llm","long-context-modeling","nlp","reasoning"],"license":"BSD-3-Clause","category":"databases-storage","readme_excerpt":"🫧 HoloBench 💿 Data • 📃 Paper Implementation of paper: Seiji Maekawa , Hayate Iso , Nikita Bhutani ( eqaul contributions). Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data 🔧 HoloBench Setup 🚀 Quick tour The following gives a quick overview of how to use HoloBench for evaluating holistic reasoning tasks. Loading Queries and Contexts First, load a sample query from the HoloBench dataset: The sample query looks like this: This snippet loads a SQL query from the wine 1 database in the HoloBench dataset. The queries dataset contains SQL-like queries used for evaluating language models. Loading the Dataset and Context Now, load the dataset and retrieve the context for the query: Executing Holistic Reasoning Finally, generate a prediction using a long-context language model: The output from the model looks like this: 📚 Citations 📜 Disclosure Embedded in, or bundled with, this product are open source software (OSS) components, datasets and other third party components identified below. The license terms respectively governing the datasets and third-party components continue to govern those portions, and you agree to those license terms, which, when applicable, specifically limit any distribution. You may receive a copy of, distribute and/or modify any open source code for the OSS component under the terms of their respective licenses, which may be BSD 3 clause license and Apache 2.0 license. In the event of conflicts between ","default_branch":null,"files":null,"tree":[],"storefront":"/r/megagonlabs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/megagonlabs/holobench/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}