{"repo":"scripbox/flume","free":true,"listed":false,"github":"https://github.com/scripbox/flume","clone":"git clone https://github.com/scripbox/flume.git","description":"A blazing fast job processing system backed by GenStage & Redis.","language":"Elixir","stars":81,"topics":["elixir-lang","genstage","background-jobs","concurrent","redis","batch-processing","scheduled-jobs","rate-limiting"],"license":null,"category":"databases-storage","readme_excerpt":"Flume Flume is a job processing system backed by GenStage & Redis Table of Contents - Features - Requirements - Installation - Usage - Pipelines - Enqueuing Jobs - Creating Workers - Scheduled Jobs - Rate Limiting - Batch Processing - Pipeline Control - Instrumentation - Writing Tests - Roadmap - References - Contributing Features - Durability - Jobs are backed up before processing. Incase of crashes, these jobs are restored. - Back-pressure - Uses gen stage to support this. - Scheduled Jobs - Jobs can be scheduled to run at any point in future. - Rate Limiting - Uses redis to maintain rate-limit on pipelines. - Batch Processing - Jobs are grouped based on size. - Logging - Provides a behaviour Flume.Logger to define your own logger module. - Pipeline Control - Queues can be pause/resume at runtime. - Instrumentation - Metrics like worker duration and latency to fetch jobs from redis are emitted via telemetry. - Exponential Back-off - On failure, jobs are retried with exponential back-off. Minimum and maximum can be set via configuration. Requirements - Elixir 1.18+ - Redis 4.0+ Installation Add Flume to your list of dependencies in mix.exs : Then run mix deps.get to install Flume and its dependencies. Usage Add Flume supervisor to your application's supervision tree: Add config/flume.exs : Import flume config in config/config.exs as given below: Pipelines Each pipeline is a GenStage pipeline having these parameters - name - Name of the pipeline queue - Name of the Redis queu","default_branch":null,"files":null,"tree":[],"storefront":"/r/scripbox","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/scripbox/flume/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}