{"repo":"Tyler-Sch/pyruhvro","free":true,"listed":false,"github":"https://github.com/Tyler-Sch/pyruhvro","clone":"git clone https://github.com/Tyler-Sch/pyruhvro.git","description":"A Rust/Python library for fast avro deserialization","language":"Rust","stars":12,"topics":["avro","deserialization","kafka","python","rust"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Ruhvro A library for deserializing schemaless avro encoded bytes into Apache Arrow record batches. This library was created as an experiment to gauge potential improvements in kafka messages deserialization speed - particularly from the python ecosystem. The main speed-ups come from: 1. A fast direct-decode/encode path that walks Avro bytes and Arrow builders together, skipping apache avro::types::Value tree construction entirely. This supports primitives, logical types (date, timestamp-millis/micros), enums, nested records, arrays, maps, and unions (2-variant nullable and N-variant sparse). Anything outside that subset transparently falls back to the original apache avro -based path. 2. Releasing Python's GIL during deserialization. 3. Multi-core processing via rayon (configurable chunk count). The speed-ups are much more noticeable on larger datasets or more complex avro schemas. Still experimental This library is still experimental and has not been tested in production. Please use with caution. Benchmarks - comparing to fastavro Apple M-series, 8 cores, processing 10,000 records using timeit (fastavro 1.12, pyarrow 24) That works out to ≈ 41× faster than fastavro on serialize and ≈ 35× on deserialize for this schema (a complex Kafka-style record with nested structs, arrays, maps, nullable fields, a multi-variant union, and an enum — see scripts/generate avro.py ). Run benchmarks locally Rust microbenchmarks ( ruhvro crate) For lower-noise measurements of the core Rust code","default_branch":null,"files":null,"tree":[],"storefront":"/r/Tyler-Sch","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Tyler-Sch/pyruhvro/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}