{"repo":"zillow/zkafka","free":true,"listed":false,"github":"https://github.com/zillow/zkafka","clone":"git clone https://github.com/zillow/zkafka.git","description":"An efficient and scalable library for stateless Kafka message processing written in Go","language":"Go","stars":70,"topics":["golang","kafka","kafka-consumer","kafka-producer","confluent-kafka","go","librdkafka"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"zkafka Install go get -u github.com/zillow/zkafka/v2 About zkafka is built to simplify message processing in Kafka. This library aims to minimize boilerplate code, allowing the developer to focus on writing the business logic for each Kafka message. zkafka takes care of various responsibilities, including: 1. Reading from the worker's configured topics 2. Managing message offsets reliably - Kafka offset management can be complex, but zkafka handles it. Developers only need to write code to process a single message and indicate whether or not it encountered an error. 3. Distributing messages to virtual partitions (details will be explained later) 4. Implementing dead lettering for failed messages 5. Providing inspectable and customizable behavior through lifecycle functions (Callbacks) - Developers can add metrics or logging at specific points in the message processing lifecycle. zkafka provides stateless message processing semantics ( sometimes, called lambda message processing). This is a churched-up way of saying, \"You write code which executes on each message individually (without knowledge of other messages)\". It is purpose-built with this type of usage in mind. Additionally, the worker implementation guarantees at least once processing (Details of how that's achieved are shown in the Commit Strategy section) NOTE : zkafka is built on top of confluent-kafka-go which is a CGO module. Therefore, so is zkafka . When building with zkafka , make sure to set CGO ENABLED=1. Feat","default_branch":null,"files":null,"tree":[],"storefront":"/r/zillow","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/zillow/zkafka/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}