{"repo":"mariomac/pipes","free":true,"listed":false,"github":"https://github.com/mariomac/pipes","clone":"git clone https://github.com/mariomac/pipes.git","description":"Complex data processing flows in Go","language":"Go","stars":17,"topics":["go","golang","etl-pipeline","etl"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"PIPES: Processing In Pipeline-Embedded Stages PIPES is a library that allows to dynamically connect multiple pipeline stages that are communicated via channels. Each stage will run in a goroutine. Its main use case is the creation of Extract-Transform-Load (ETL) components, such as data cleaners or monitoring agents. API doc: https://pkg.go.dev/github.com/mariomac/pipes Tutorial: https://github.com/mariomac/pipes/tree/main/tutorial It is a fork version of the Red Hat's & IBM Gopipes library, but this library is not related to any concrete product. This library allows wrapping functions within Nodes of a connected Graph. In order to pass data across the nodes, each wrapped function must receive, as arguments, an input channel, an output channel, or both. There are three types of nodes: Start node: each of the starting point of a graph. This is, all the nodes that bring information from outside the graph: e.g. because they generate them or because they acquire them from an external source like a Web Service. A graph must have at least one Start node. A Start node must have at least one output node. Middle node: any intermediate node that receives data from another node, processes/filters it, and forwards the data to another node. A Middle node must have at least one output node. Final node: any node that receives data from another node and does not forward it to another node, but can process it and send the results to outside the graph (e.g. memory, storage, web...) Nodes are i","default_branch":null,"files":null,"tree":[],"storefront":"/r/mariomac","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mariomac/pipes/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}