{"repo":"mxmlnkn/rapidgzip","free":true,"listed":false,"github":"https://github.com/mxmlnkn/rapidgzip","clone":"git clone https://github.com/mxmlnkn/rapidgzip.git","description":"Gzip Decompression and Random Access for Modern Multi-Core Machines","language":"Python","stars":463,"topics":["cli","cpp","cpp17","cpp17-library","decompression","gzip","gzip-decompression","library","parallel","python-library"],"license":"Apache-2.0","category":"cli-tools","readme_excerpt":"Rapidgzip: Parallelized Decompression of Gzip Files with Support for Fast Random Access This repository contains the command line tool rapidgzip , which can be used for parallel decompression of almost any gzip file. Other tools, such as bgzip, can only parallelize decompression of gzip files produced by themselves. rapidgzip works with all files, especially those produced by the usually installed GNU gzip. How this works can be read in the pugz paper or in the rapidgzip paper, which builds upon the former. The Python module provides a RapidgzipFile class, which can be used to seek inside gzip files without having to decompress them first. Alternatively, you can use this simply as a parallelized gzip decoder as a replacement for Python's builtin gzip module in order to fully utilize all your cores. The random seeking support is the same as provided by indexed gzip, but further speedups are realized at the cost of higher memory usage thanks to a least-recently-used cache in combination with a parallelized prefetcher. This repository is a light-weight fork of the indexed bzip2 repository, in which the main development takes place. This repository was created for visibility reasons and in order to keep indexed bzip2 and rapidgzip releases and packaging separate. It will be updated at least for each release. Issues regarding rapidgzip should be opened here. Table of Contents 1. Installation 2. Performance 1. Scaling Benchmarks on 2xAMD EPYC CPU 7702 (2x64 cores) 2. Scaling Benchm","default_branch":null,"files":null,"tree":[],"storefront":"/r/mxmlnkn","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mxmlnkn/rapidgzip/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}