{"repo":"mxmlnkn/librapidarchive","free":true,"listed":false,"github":"https://github.com/mxmlnkn/librapidarchive","clone":"git clone https://github.com/mxmlnkn/librapidarchive.git","description":"Fast parallel random access to bzip2 and gzip files in Python","language":"C++","stars":87,"topics":["bzip2","cpp","cpp17-library","decompression","parallel","python","library","gzip","random-access","cli"],"license":"Apache-2.0","category":"cli-tools","readme_excerpt":"Parallel Random Access to bzip2, gzip (and hopefully more in the future) This repository contains the code for the indexed bzip2 and rapidgzip Python modules. Both are built upon the same basic architecture to enable block-parallel decoding based on prefetching and caching. rapidgzip This module provides: - a rapidgzip command line tool for parallel decompression of gzip files with a similar command line interface to gzip so that it can be used as a replacement. - a rapidgzip.open Python method for reading and seeking inside gzip files using multiple threads for a speedup of 21 over the built-in gzip module using a 12-core processor. The random seeking support is similar to the one provided by indexed gzip, and the parallel capabilities are effectively a working version of pugz, which is only a concept and only works with a limited subset of file contents, namely non-binary (ASCII characters 0 to 127) compressed files. Module Bandwidth / (MB/s) Speedup ------------------------------------- -------------------- --------- gzip 250 1 rapidgzip with parallelization = 1 488 1.9 rapidgzip with parallelization = 2 902 3.6 rapidgzip with parallelization = 12 4463 17.7 rapidgzip with parallelization = 24 5240 20.8 See here for the dedicated repository and ReadMe. A paper describing the implementation details and showing the scaling behavior with up to 128 cores has been accepted in ACM HPDC'23, The 32nd International Symposium on High-Performance Parallel and Distributed Computing. If","default_branch":null,"files":null,"tree":[],"storefront":"/r/mxmlnkn","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mxmlnkn/librapidarchive/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}