{"repo":"RTBHOUSE/kafka-workers","free":true,"listed":false,"github":"https://github.com/RTBHOUSE/kafka-workers","clone":"git clone https://github.com/RTBHOUSE/kafka-workers.git","description":"Kafka Workers is a client library which unifies records consuming from Kafka and processing them by user-defined WorkerTasks.","language":"Java","stars":32,"topics":["kafka","kafka-workers","kafka-consumer","stream-processing","multithreading","partitioning","backpressure","parallel","parallel-consumer"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Kafka Workers Kafka Workers is a client library which unifies records consuming from Kafka and processing them by user-defined tasks. It provides: - higher level of distribution because of sub-partitioning defined by WorkerPartitioner , - tighter control of offsets commits to Kafka applied by RecordStatusObserver , - possibility to pause and resume processing by WorkerTask for given partition, - at-least-once state and output semantics, - backpressure, - processing timeouts, - handling failures. Motivation The major thing to address was a threading model with better resources utilization in terms of: - decoupling consumption and processing with records buffering (for better performance) - higher level of distribution for processing (higher than consumption and not always limited by partitions count) Especially for the second point we have a repeating scenario: for some reasons we want to keep the same (not too high) count of partitions which gives us a good distribution and efficiency of day-to-day processing. The problem occurs when we need to (re)process the stream with a huge (consumer) lag (for example because of some failure). In such a scenario we would like to temporarily bump up the number of partitions for data stored in Kafka to better utilize cpu and/or external APIs. It is what we called sub-partitioning. What is important in terms of proper distributed processing, stream of records from one TopicPartition could be reordered now but records with the same WorkerSub","default_branch":null,"files":null,"tree":[],"storefront":"/r/RTBHOUSE","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/RTBHOUSE/kafka-workers/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}