{"repo":"SpaceCell/lightstream","free":true,"listed":false,"github":"https://github.com/SpaceCell/lightstream","clone":"git clone https://github.com/SpaceCell/lightstream.git","description":"Fast data transport : Apache Arrow, Protobuf and Message Pack data on the one stream. Measured faster than Arrow Flight on all open benchmarks. HTTP, TCP, QUIC, Websocket, Webtransport, Stdio, UDS, all interchangeable, with 64-byte SIMD compatibilty maintained over the network and to/from disk via custom Arrow Mmap and Parquet Readers and Writers","language":"Rust","stars":40,"topics":["apache-arrow","arrow-flight","arrow-ipc","arrow-rs","data","data-engineering","data-science","fast","low-latency","low-latency-streaming"],"license":null,"category":"data-pipelines","readme_excerpt":"Lightstream Send and receive typed Arrow and Protobuf data easily across networks and processes at speed. Supported URI schemes include: Transport URI --------------------- ----------------------- TCP tcp://host:port Unix domain socket uds:///path/to/socket WebSocket ws://host/path Secure WebSocket wss://host/path HTTP http://host/path HTTPS https://host/path QUIC quic://host:port WebTransport wt://host/path Standard input/output stdio:// Installation Rust Python See the Python README and Rust README for setup details. Quick Start Rust Send Tables Receive Tables Lightstream protocol Multiplex Protobuf messages and Arrow tables on the one connection. Memory-mapped Arrow IPC Python Read and writing Hit your favourite tools Every reader implements the Arrow PyCapsule stream protocol, so Arrow-compatible libraries can consume a Lightstream reader directly. Protobuf and Arrow moving across process The problem TLDR : At the time of writing, there wasn't a (at least well-known) 'all-in-one' data transport that makes it effortless to send data and metadata at highly optimised speeds between services and processes, whilst maintaining a dual-sided typing contract compatible with the major data ecosystem. Movement Friction : Right now, moving Arrow data (the common tabular interface standard) between high-throughput services is not as easy as it could be. You pick a transport such as gRPC, HTTP, TCP or WebSocket. If strongly typed metadata like Protobuf needs to accompany the Arrow data","default_branch":null,"files":null,"tree":[],"storefront":"/r/SpaceCell","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/SpaceCell/lightstream/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}