{"repo":"go-pkgz/flow","free":true,"listed":false,"github":"https://github.com/go-pkgz/flow","clone":"git clone https://github.com/go-pkgz/flow.git","description":"Package flow provides support for basic FBP / pipelines","language":"Go","stars":26,"topics":["golang","go","library","flow","pipeline","fbp","etl"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Flow - FBP / pipelines / workers pool Package flow provides support for very basic FBP / pipelines. It helps to structure multistage processing as a set of independent handlers communicating via channels. The typical use case is for ETL (extract, transform, load) type of processing. Package flow doesn't introduce any high-level abstraction and keeps everything in the hand of the user. Package pool provides a simplified version of flow suitable for cases with a single-handler flows. Details about flow package - Each handler represents an async stage. It consumes data from an input channel and publishes results to an output channel. - Each handler runs in a separate goroutine. - User must implement Handler functions and add it to the Flow. - Each handler usually creates an output channel, reads from the input channel, processes data, sends results to the output channel and closes the output channel. - Processing sequence determined by the order of those handlers. - Any Handler can run in multiple concurrent goroutines (workers) by using the Parallel decorator. - FanOut allows to pass multiple handlers in broadcast mode, i.e., each handler gets every input record. Outputs from these handlers merged into single output channel. - Processing error detected as return error value from user's handler func. Such error interrupts all other running handlers gracefully and won't keep any goroutine running/leaking. - Each Flow object can be executed only once. - Handler should handle conte","default_branch":null,"files":null,"tree":[],"storefront":"/r/go-pkgz","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/go-pkgz/flow/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}