{"repo":"Noorts/PDXearch","free":true,"listed":false,"github":"https://github.com/Noorts/PDXearch","clone":"git clone https://github.com/Noorts/PDXearch.git","description":"[WIP] A state-of-the-art IVF index for lightweight but fast (filtered) vector similarity search in DuckDB.","language":"C++","stars":13,"topics":["duckdb","duckdb-extension","pdx","vector-similarity-search","vss","pdxearch"],"license":"MIT","category":"data-pipelines","readme_excerpt":"The PDXearch DuckDB Extension A state-of-the-art IVF index for lightweight but fast (filtered) vector similarity search. - Why PDXearch? - Install - Usage - Getting Started - Configuration - Index Creation - Index Search - Index Metadata - Limitations - Acknowledgements - License Why PDXearch? DuckDB offers vector similarity search (VSS) out of the box, through its fixed-size ARRAY column type and distance functions (docs). These functions return exact results, but are often too slow on large datasets. The official DuckDB VSS extension introduces a graph-based (HNSW) VSS index. You can create this index on your table to speed up the vector search queries. Unfortunately, although these graph-based indexes deliver fast search, they take up a considerable amount of memory and take long to construct. The PDXearch extension aims to address these drawbacks. It achieves competitive search performance, while using less memory and being significantly faster to construct. This is made possible by a state-of-the-art partition-based (IVF) index. To be precise, we rely on the CWI's PDX data layout and the accompanying search framework called PDXearch. Furthermore, this extension integrates tightly with DuckDB's internals to parallelize across row groups, allowing us to squeeze more performance out of modern hardware. Install [!WARNING] The extension is unstable and experimental. We're actively working on adding features and improving stability. The extension will be made available as a co","default_branch":null,"files":null,"tree":[],"storefront":"/r/Noorts","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Noorts/PDXearch/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}