{"repo":"spreedly/kaffe","free":true,"listed":false,"github":"https://github.com/spreedly/kaffe","clone":"git clone https://github.com/spreedly/kaffe.git","description":"An opinionated Elixir wrapper around brod, the Erlang Kafka client, that supports encrypted connections to Heroku Kafka out of the box.","language":"Elixir","stars":165,"topics":["elixir","kafka"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Kaffe An opinionated, highly specific, Elixir wrapper around Brod: the Erlang Kafka client. :coffee: NOTE : Although we're using this in production at Spreedly it is still under active development. The API may change and there may be serious bugs we've yet to encounter. Table of Contents generated with DocToc - Installation - Kaffe Consumer Usage - Kaffe GroupMember - Batch Message Consumer - Managing how offsets are committed - Kaffe Consumer - Single Message Consumer (Deprecated) - async message acknowledgement - Kaffe Producer Usage - Heroku Configuration - Producing to Kafka - Compression Config - Testing - Setup - Running Installation 1. Add kaffe to your list of dependencies in mix.exs : 2. If using crc32cer version 1.0, install cmake. 3. Configure a Kaffe Consumer and/or Producer Kaffe Consumer Usage Consumers receive a list of messages and work as part of the :brod group member behavior. This has a few important benefits: 1. Group members assign a \"subscriber\" to each partition in the topic. Because Kafka topics scale with partitions, having a worker per partition usually increases throughput. 2. Group members correctly handle partition assignments across multiple clients in a consumer group. This means that this mode of operation will scale horizontally (e.g., multiple dynos on Heroku). 3. Downstream processing that benefits from batching (like writing to another Kafka topic) is more easily supported. There is also legacy support for single message consumers, which p","default_branch":null,"files":null,"tree":[],"storefront":"/r/spreedly","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/spreedly/kaffe/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}