{"repo":"loicalleyne/quacfka","free":true,"listed":false,"github":"https://github.com/loicalleyne/quacfka","clone":"git clone https://github.com/loicalleyne/quacfka.git","description":"Go library to stream Kafka protobuf messages to DuckDB","language":"Go","stars":26,"topics":["arrow","duckdb","golang","golang-package","kafka","protobuf"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Quacfka 🏹🦆 =================== Go library to stream Kafka protobuf messages to DuckDB. Uses generics. Use your protobuf message as a type parameter to autogenerate an Arrow schema, provide a protobuf unmarshaling func, and stream data into DuckDB with a very high throughput. Features Arrow schema generation from a protobuf message type parameter - Converts a proto.Message into an Apache Arrow schema - Supports nested types Configurable loggers - Set Debug, Error, and benchmark loggers 🚀 Install Using Quacfka is easy. First, use go get to install the latest version of the library. 💡 Usage You can import quacfka using: Create a new Orchestrator, configure the Kafka client, processing and DuckDB, then Run(). Kafka client can be configured with a slice of franz-go/pkg/kgo.Opt or SASL user/pass auth. Generate random data to emulate the Kafka topic 💫 Show your support Give a ⭐️ if this project helped you! Feedback and PRs welcome. Licence Quacfka is released under the Apache 2.0 license. See LICENCE","default_branch":null,"files":null,"tree":[],"storefront":"/r/loicalleyne","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/loicalleyne/quacfka/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}