{"repo":"SpaceCell/minarrow","free":true,"listed":false,"github":"https://github.com/SpaceCell/minarrow","clone":"git clone https://github.com/SpaceCell/minarrow.git","description":"Fast, minimal, and ergonomic zero-copy Apache Arrow tabular data implementation in Rust, with PyO3, Python bindings, NdArray, XArray, shared-memory and low-latency SIMD for data-engineering and columnar workloads. Bridge to Polars and Arrow-rs over FFI effortlessly.","language":"Rust","stars":99,"topics":["arrow","data-science","polars","rust","apache","data","data-engineering","engineering","low-latency","simd"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Fast, minimal, and ergonomic Apache Arrow implementation in Rust, with PyO3 and Python bindings. Highlights : - i8 through u64 FloatArray f32, f64 StringArray UTF-8 with u32 or u64 offsets BooleanArray Bit-packed with null mask CategoricalArray Dictionary-encoded DatetimeArray Timestamps, dates, durations Semantic groupings ( NumericArray , TextArray , TemporalArray ) support flexibility call-site signatures and static dispatch. Array and Table ( RecordBatch ) sit on top, with chunked Super ( Vec ) versions for streaming. Bonus LAPACK-compatible Matrix and Cube types support analytical workload variations. NdArray n-dimensional numeric data, with chunked ( SuperNdArray ) and labelled ( XArray ) forms, and hands tensors to PyTorch, JAX, and NumPy zero-copy over DLPack. Zero-Copy Views Ergonomic zero-copy row and column selection. Off-the-Wire pattern Arrow Interop Embed Python in Rust. Bench in /examples. Normal Rust - use Polars, Arrow, convert at the boundary, stay native internally: Architecture Minarrow uses enums for type dispatch instead of trait objects. Benchmarks Array performance Sum of 1,000 integers, averaged over 1,000 runs¹: Implementation Time ---------------- ------ Raw Vec 85 ns Minarrow IntegerArray (direct) 88 ns Minarrow IntegerArray (via enum) 124 ns Arrow-rs Int64Array (struct) 147 ns Arrow-rs Int64Array (dyn) 181 ns Minarrow's direct access is within the noise threshold of raw Vec performance whilst maintaining SIMD-compatible alignment. Python Roundtrip","default_branch":null,"files":null,"tree":[],"storefront":"/r/SpaceCell","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/SpaceCell/minarrow/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}