{"repo":"beir-cellar/beir","free":true,"listed":false,"github":"https://github.com/beir-cellar/beir","clone":"git clone https://github.com/beir-cellar/beir.git","description":"A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.","language":"Python","stars":2272,"topics":["nlp","information-retrieval","bert","benchmark","sentence-transformers","question-generation","retrieval","passage-retrieval","elasticsearch","dpr"],"license":"Apache-2.0","category":"machine-learning","readme_excerpt":"Paper Installation Quick Example Datasets Wiki Hugging Face :beers: What is it? BEIR is a heterogeneous benchmark containing diverse IR tasks. It also provides a common and easy framework for evaluation of your NLP-based retrieval models within the benchmark. For an overview , checkout our new wiki page: https://github.com/beir-cellar/beir/wiki. For models and datasets , checkout out Hugging Face (HF) page: https://huggingface.co/BeIR. For more information, checkout out our publications: - BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (NeurIPS 2021, Datasets and Benchmarks Track) - Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard (SIGIR 2024 Resource Track) :beers: Installation Install via pip: If you want to build from source, use: Tested with python versions 3.9+ :beers: Features - Preprocess your own IR dataset or use one of the already-preprocessed 17 benchmark datasets - Wide settings included, covers diverse benchmarks useful for both academia and industry - Evaluates well-known retrieval architectures (lexical, dense, sparse and reranking-based) - Add and evaluate your own model in a easy framework using different state-of-the-art evaluation metrics :beers: Quick Example For other example codes, please refer to our Examples and Tutorials Wiki page. Quick Example with Sentence-BERT Quick Example with LoRA & vLLM First install peft, vllm & accelerate using the following installations: Quick Ex","default_branch":null,"files":null,"tree":[],"storefront":"/r/beir-cellar","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/beir-cellar/beir/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}