{"repo":"immobiliare/ufoid","free":true,"listed":false,"github":"https://github.com/immobiliare/ufoid","clone":"git clone https://github.com/immobiliare/ufoid.git","description":"Ultra Fast Optimized Image Deduplication.","language":"Python","stars":28,"topics":["automation","computer-vision","deduplication","images","python","immobiliare-labs"],"license":"MIT","category":"machine-learning","readme_excerpt":"UFOID Ultra Fast Optimized Image Deduplication. Table of Contents - Introduction - Installation - Configuration - Results - Benchmarks - Changelog - Contributing - Powered Apps - Support Introduction The goal of this project is to efficiently detect and handle duplicate images within a dataset and across different datasets. The code uses perceptual hashing (image hashing) to convert images into hash representations, allowing for quick comparison and identification of duplicate images based on a specified distance threshold. The project provides two main functionalities: 1. Duplicate detection within a single dataset using chunks: this method processes the dataset in smaller chunks to optimize performance for large datasets. 2. Duplicate detection between two datasets using chunks: this method allows for comparison between a reference dataset and a new dataset to identify any overlapping duplicate images. The competitor imagededup library, running on hash sizes of 8 bit, demonstrates fast degradation of accuracy when increasing threshold, and shows heavy degradation of computation time when duplicates increase for false positives. In contrast, our library, shows consistent results over higher threshold (and this can be important to detect near-duplicates as shown in the accuracy benchmark). Furthermore, despite running on 16-bit hashes, the strong computation optimization allow our library to be faster in a consistent way (more than double speed on 100k datasets). More details","default_branch":null,"files":null,"tree":[],"storefront":"/r/immobiliare","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/immobiliare/ufoid/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}