{"repo":"princeton-ddss/lsh","free":true,"listed":false,"github":"https://github.com/princeton-ddss/lsh","clone":"git clone https://github.com/princeton-ddss/lsh.git","description":"DuckDB community extension for locality-sensitive hashing (LSH)","language":"Rust","stars":14,"topics":["approximate-nearest-neighbor-search","data-matching","deduplication","duckdb","duckdb-community","duckdb-extension","entity-resolution","fuzzy-matching","lsh","record-linkage"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Locality-Sensitive Hashing (LSH) DuckDB Extension DuckDB extension for locality-sensitive hashing (LSH), using the Rust implementations from the zoomerjoin R package. (For a conceptual review and a description of that package, see https://doi.org/10.21105/joss.05693.) Installation lsh is a DuckDB Community Extension. It can be installed and loaded in DuckDB like so: Available Functions 1. MinHash a. Text Input: f(VARCHAR, INT, INT, INT, INT) → LIST(UINT64 or UINT32) - 64-bit: lsh min(string, ngram width, band count, band size, seed) - 32-bit: lsh min32(string, ngram width, band count, band size, seed) b. Custom Shingle Set Input: f(LIST(VARCHAR), INT, INT, INT) → LIST(UINT64 or UINT32) - 64-bit: lsh min(shingles, band count, band size, seed) - 32-bit: lsh min32(shingles, band count, band size, seed) 2. Euclidean Hashing: f(ARRAY(DOUBLE), DOUBLE, INT, INT, INT) → LIST(UINT64 or UINT32) - 64-bit: lsh euclidean(coordinate array, bucket width, band count, band size, seed) - 32-bit: lsh euclidean32(coordinate array, bucket width, band count, band size, seed) 3. Jaccard Similarity: f(VARCHAR, VARCHAR, INT) → DOUBLE - lsh jaccard(string left, string right, ngram width) Suggested Usage We do not recommend creating and storing the full ARRAY::[band count] -type columns, as they become large very quickly. Instead, we recommend generating bands on-the-fly in join conditions (i.e., when generating comparisons/potential matches). This reduces storage needs and memory consumption. Further,","default_branch":null,"files":null,"tree":[],"storefront":"/r/princeton-ddss","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/princeton-ddss/lsh/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}