{"repo":"definite-app/duck-takes-flight","free":true,"listed":false,"github":"https://github.com/definite-app/duck-takes-flight","clone":"git clone https://github.com/definite-app/duck-takes-flight.git","description":"A high-performance data streaming system using DuckDB and Apache Arrow Flight.","language":"Python","stars":96,"topics":["arrow","duckdb","python"],"license":null,"category":"data-pipelines","readme_excerpt":"Duck Takes Flight A high-performance data streaming system using DuckDB and Apache Arrow Flight. Ok, now do it in Rust Components 1. DuckDB Flight Server ( duckdb flight server.py ): A server that exposes DuckDB through Arrow Flight protocol 2. Data Loader ( load data.py ): Continuously generates and loads random data into DuckDB 3. Query Client ( query data.py ): Executes continuous queries against the loaded data Setup 1. Install dependencies: 2. Start the server: 3. Start the data loader: 4. Start the query client: Data Schema The system creates a table called concurrent test with the following schema: - batch id : BIGINT - timestamp : VARCHAR - value : DOUBLE - category : VARCHAR Features - Persistent storage using DuckDB - High-performance data transfer using Arrow Flight - Continuous data loading and querying - Memory-efficient batch processing - Aligned Arrow buffers for optimal performance","default_branch":null,"files":null,"tree":[],"storefront":"/r/definite-app","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/definite-app/duck-takes-flight/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}