{"repo":"naughtygopher/nibbler","free":true,"listed":false,"github":"https://github.com/naughtygopher/nibbler","clone":"git clone https://github.com/naughtygopher/nibbler.git","description":"A light-weight package for in-app micro batch processing","language":"Go","stars":17,"topics":["batch","batch-processing","event-processing","go","go-library","go-package","golang","micro-batches","microbatch","stream"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Nibbler Nibbler is a resilient & minimal package which helps you implement micro-batch processing, within an application . Nibbler remains minimal with its 0 external dependencies and remains resilient within the context of the application by gracefully handling errors and panics. IMPORTANT: This is not a general purpose distributed task queue. What is Micro-batch Processing? Micro-batch processing is a way to handle data by breaking a big task into smaller pieces and processing them one piece at a time. This method is useful in processing realtime data or stream processing, where the incoming data is split into \"micro-batches\" and processed quickly, rather than waiting to collect all data at once or process one by one. The same concept can also be extended to handle events processing. So, we have a queue subscriber, and instead of processing the events individually, we create micro batches and process them. The processing of a single micro batch can be triggered in two ways, based on a time ticker or if the micro batch size is full. i.e. process a non empty batch if duration X has passed or if the batch size is full Why use nibbler? High-throughput event and stream processing systems benefit significantly from batch processing rather than handling events individually. Well-implemented batching optimizes resource utilization across downstream dependencies like databases, APIs, and external systems-by dramatically reducing IOPS. Batch operations enable more efficient use of ne","default_branch":null,"files":null,"tree":[],"storefront":"/r/naughtygopher","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/naughtygopher/nibbler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}