{"repo":"deric/es-dedupe","free":true,"listed":false,"github":"https://github.com/deric/es-dedupe","clone":"git clone https://github.com/deric/es-dedupe.git","description":"Tool for removing duplicate documents from Elasticsearch","language":"Python","stars":54,"topics":["elasticsearch","duplicates","duplicity"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"ES-dedupe A tool for removing duplicated documents that are grouped by some unique field (e.g. --field Uuid ). Usage Use -h/--help to see supported options: Remove duplicates from index exact-index-name while searching for unique Uuid field: Multiple unique fields Build a local index using md5(time,device id) as an unique key. It might require a significant amount of memory (depends on the size of your index, it can easily grow to gigabytes - it's stored as a Python dict with a string key, which might occupy a large amount of memory). Examples More advanced example with documents containing timestamps. WARNING: Running huge bulk operations on Elastic cluster might influence performance of your cluster or even crash some nodes if heap is not large enough. A sliding window -w / --window could be used to prevent running out of memory on larger indexes (if you have a timestamp field): Requirements For the installation use the tools provided by your operating system. On Linux this can be one of the following: yum, dnf, apt, yast, emerge, .. Install python (2 or 3, both will work) Install python ujson and python requests for the fitting python version On Windows you are pretty much on your own, but fear not, you can do the following ;-) Download and install a python version from https://www.python.org/ . Open a console terminal and head to the repository copy of es-deduplicator, then run: pip install -r requirements.txt Testing Test can be run from a Docker container. You can use s","default_branch":null,"files":null,"tree":[],"storefront":"/r/deric","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/deric/es-dedupe/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}