{"repo":"softwaremill/kmq","free":true,"listed":false,"github":"https://github.com/softwaremill/kmq","clone":"git clone https://github.com/softwaremill/kmq.git","description":"Kafka-based message queue","language":"Scala","stars":341,"topics":["kafka","message-queue","reactive-streams","java","scala","cats-effect"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Kafka Message Queue Using kmq you can acknowledge processing of individual messages in Kafka, and have unacknowledged messages re-delivered after a timeout. This is in contrast to the usual Kafka offset-committing mechanism, using which you can acknowledge all messages up to a given offset only. If you are familiar with Amazon SQS, kmq implements a similar message processing model. How does this work? For a more in-depth overview see the blog: Using Kafka as a message queue, and for performance benchmarks: Kafka with selective acknowledgments (kmq) performance & latency benchmark The acknowledgment mechanism uses a marker topic, which should have the same number of partitions as the \"main\" data topic (called the queue topic). The marker topic is used to track which messages have been processed, by writing start/end markers for every message. Using kmq An application using kmq should consist of the following components: a number of RedeliveryTracker s. This components consumes the marker topic and redelivers messages if appropriate. Multiple copies should be started in a cluster for fail-over. Uses automatic partition assignment. components which send data to the queue topic to be processed queue clients, either custom or using the KmqClient Maven/SBT dependency SBT: \"com.softwaremill.kmq\" %% \"core\" % \"0.3.1\" Maven: com.softwaremill.kmq core 2.13 0.3.1 Note: The supported Scala versions are: 2.12, 2.13. Client flow The flow of processing a message is as follows: 1. read messag","default_branch":null,"files":null,"tree":[],"storefront":"/r/softwaremill","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/softwaremill/kmq/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}