{"repo":"moj-analytical-services/splink","free":true,"listed":false,"github":"https://github.com/moj-analytical-services/splink","clone":"git clone https://github.com/moj-analytical-services/splink.git","description":"Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends","language":"Python","stars":2345,"topics":["record-linkage","spark","em-algorithm","deduplication","deduplicate-data","entity-resolution","data-matching","fuzzy-matching","data-science","duckdb"],"license":"MIT","category":"databases-storage","readme_excerpt":"[!IMPORTANT] 🎉 Splink 4 has been released! Examples of new syntax are here and a release announcement is here. Fast, accurate and scalable data linkage and deduplication Splink is a Python package for probabilistic record linkage (entity resolution) that allows you to deduplicate and link records from datasets that lack unique identifiers. It is used widely by within government, academia and the private sector - see use cases. Key Features ⚡ Speed: Capable of linking a million records on a laptop in around a minute. 🎯 Accuracy: Support for term frequency adjustments and user-defined fuzzy matching logic. 🌐 Scalability: Execute linkage in Python (using DuckDB) or big-data backends like Spark for 100+ million records. 🎓 Unsupervised Learning: No training data is required for model training. 📊 Interactive Outputs: A suite of interactive visualisations help users understand their model and diagnose problems. Splink's linkage algorithm is based on Fellegi-Sunter's model of record linkage, with various customisations to improve accuracy. What does Splink do? Consider the following records that lack a unique person identifier: Splink predicts which rows link together: and clusters these links to produce an estimated person ID: What data does Splink work best with? Splink performs best with input data containing multiple columns that are not highly correlated . For instance, if the entity type is persons, you may have columns for full name, date of birth, and city. If the entity","default_branch":null,"files":null,"tree":[],"storefront":"/r/moj-analytical-services","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/moj-analytical-services/splink/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}