{"repo":"tolitius/grete","free":true,"listed":false,"github":"https://github.com/tolitius/grete","clone":"git clone https://github.com/tolitius/grete.git","description":"kafka client with threads and a scheduler","language":"Clojure","stars":16,"topics":["kafka","producer","consumer","concurrency","thread-pool"],"license":null,"category":"data-pipelines","readme_excerpt":"grete is gregor's sister that adds a threadpool and a scheduler ... and some Java API ... and the latest kafka (at the moment of writing) the idea behind grete is to be able to start a farm of kafka consumers that listen to (potentially) multiple topics and apply a simple consuming function. - spilling the beans - produce - consume - callbacks - stream it - Java API - several topics at once - License spilling the beans it is quite common for the same app to produce and consume, hence we'll use one config for producing and consuming: produce produce a couple of messages (to foos topic): consume a sample consuming function \" process \": start a farm of consumers ( 42 threads as per config): once the \"farm\" is started you'll see those two messages that were produces above: values here are strings, but could be byte arrays given bytearray de/serializers. as with other thread pools, it's a good idea to shut them down once we done working with them: callbacks a kafka producer has an internal accumulator (kept in Deque) where it stores all the events before sending them out to the server (a.k.a. broker). when it is ready to send them, it splits the events stored in the accumulator in batches (controlled by a \" batch.size \" prop) and sends them out batch by batch. the wait time before the actual \"publish send\" is controlled via a \" linger.ms \" producer configuration property that maintains the balance of latency vs. throughput. hence the kafka publishing process is asynchronous by des","default_branch":null,"files":null,"tree":[],"storefront":"/r/tolitius","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/tolitius/grete/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}