{"repo":"zerodha/dungbeetle","free":true,"listed":false,"github":"https://github.com/zerodha/dungbeetle","clone":"git clone https://github.com/zerodha/dungbeetle.git","description":"A distributed job server built specifically for queuing and executing heavy SQL read jobs asynchronously. Separate out reporting layer from apps. MySQL, Postgres, ClickHouse.","language":"Go","stars":1263,"topics":["job-scheduler","job-queue","sql-query","reporting","sql","database","workers","mysql","postgres","postgresql"],"license":"MIT","category":"databases-storage","readme_excerpt":"DungBeetle is a lightweight, single binary distributed job server designed for queuing and asynchronously executing large numbers of SQL read jobs (eg: reports) against SQL databases. When the read jobs are executed, the results are written to separate ephemeral results databases (where the results of every job is its own dedicated table), enabling faster retrieval. A prominent usecase is user facing report generation in applications where requests can be queued and reports returned asynchronously without overloading large source databases. Features - Supports MySQL, PostgreSQL, ClickHouse as source databases. - Supports MySQL and PostgreSQL as result / cache databases for job results. - Standalone server that exposes HTTP APIs for managing jobs and groups of jobs (list, post, status check, cancel jobs). - Reads SQL queries from .sql files and registers them as jobs ready to be queued. - Supports multi-process, multi-threaded, asynchronous distributed job queueing via a common broker backend (Redis, AMQP etc.) Usecase Consider an application with a very large SQL database. When there are several thousand concurrent users requesting reports from an application connected to it, every second of I/O delay during query execution can bottleneck the application and the database, causing a snowball effect. Instead, user requests for report generations can be deferred to a job queue in the backend, there by immediately freeing up the frontend application. The reports are presented to ","default_branch":null,"files":null,"tree":[],"storefront":"/r/zerodha","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/zerodha/dungbeetle/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}