{"repo":"jankovicsandras/plpgsql_bm25","free":true,"listed":false,"github":"https://github.com/jankovicsandras/plpgsql_bm25","clone":"git clone https://github.com/jankovicsandras/plpgsql_bm25.git","description":"BM25 search implemented in PL/pgSQL","language":"Jupyter Notebook","stars":125,"topics":["bm25","document-search","okapi","plpgsql","postgres","postgresql","search","text-search","bm25okapi"],"license":"Unlicense","category":"databases-storage","readme_excerpt":"README.md plpgsql bm25 BM25 search implemented in PL/pgSQL ---- News - many BM25 algorithms are implemented, some rank bm25 compatible and some extra ( '' (Okapi), 'l', 'plus', 'robertson', 'luceneaccurate', 'atire', 'bm25l', 'bm25plus' ) - stopword filter in many languages (en, fr, es, pt, it, de, nl, sv, no, nn, da, ru, fi, hu, ga, id) Other Features - hybrid search with pgvector and Reciprocal Rank Fusion, see plpgsql bm25rrf.sql, example: Postgres hybrid search RRF.ipynb - an optimized Python implementation is available, see BM25opt, which runs 30-40 x faster than rank bm25 Roadmap / TODO - better documentation - tokenization options? - bm25scoressum() temp table? - implement other algorithms, e.g. BMX ? ---- Example usage: 1. download and execute plpgsql bm25.sql to load the functions, e.g. 2. then BM25Okapi is the default algoritm. The parameter can select the following algorithms: : Stopword filtering. The following language tags can be selected: Empirical results show that and active stopword filtering (e.g. ) usually lead to better results. Calling these from Python with a simple psycopg2 helper: ---- API - This creates the BM25 index by creating these new tables: - The index creation is a costy operation, but required after every change in the corpus (the original texts in tablename- columnname). - values: is BM25Okapi (default), . - values: is no stopword filtering (default), - This is the search function, which returns the top documents and their scores that are m","default_branch":null,"files":null,"tree":[],"storefront":"/r/jankovicsandras","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jankovicsandras/plpgsql_bm25/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}