{"repo":"snap-stanford/stark","free":true,"listed":false,"github":"https://github.com/snap-stanford/stark","clone":"git clone https://github.com/snap-stanford/stark.git","description":"(NeurIPS D&B 2024) STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases","language":"Python","stars":335,"topics":["llm","multimodal","graph","nlp","knowledge-base","information-retrieval","semi-structured-data"],"license":"MIT","category":"ai-agents","readme_excerpt":"STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases NEWS - [Oct 2024] Leaderboard: We construct the official leaderboard on Hunggingface and we are calling for submissions! - [Oct 2024] Camera-ready paper is out! We add multiple retrieval models including BM25, Colbertv2, GritLM. - [Sep 2024] STaRK is accepted to 2024 NeurIPS Dataset & Benchmark Track! - [Jun 2024] We make our benchmark as a pip package stark-qa. You can directly load the data from the package now! - [Jun 2024] We migrate our data to Hugging Face! You don't need to change anything, the data will be automatically downloaded. - [May 2024] We have augmented our benchmark with three high-quality human-generated query datasets which are open to access. See more details in our updated arxiv! - [May 9th 2024] We release STaRK SKB Explorer, an interactive interface for you to explore our knowledge bases! - [May 7th 2024] We present STaRK in the 2024 Stanford Annual Affiliates Meeting and 2024 Stanford Data Science Conference. - [May 5th 2024] STaRK was reported on Marketpost and 智源社区 BAAI. Thanks for writing about our work! - [Apr 21st 2024] We release the STaRK benchmark. What is STaRK? STaRK is a large-scale Semi-structured Retrieval Benchmark on Textual and Relational Knowledge bases, covering applications in product search, academic paper search, and biomedicine inquiries. Featuring diverse, natural-sounding, and practical queries that require context-specific reasoning, STaRK sets a new s","default_branch":null,"files":null,"tree":[],"storefront":"/r/snap-stanford","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/snap-stanford/stark/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}