{"repo":"datayoga-io/datayoga","free":true,"listed":false,"github":"https://github.com/datayoga-io/datayoga","clone":"git clone https://github.com/datayoga-io/datayoga.git","description":"streaming data pipeline platform","language":"Python","stars":30,"topics":["etl","pipeline","transformation","cdc","sqlalchemy","kafka","singer","elt","data","database"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Introduction DataYoga is a framework for building and running streaming or batch data pipelines. DataYoga uses low-code to easily define data pipelines using a declarative markup language using YAML files. Concepts Job - A Job is composed of a series of Steps that reads information from a source, performs transformations, and write to a target. Many sources and targets are supported, including relational databases, non relational databases, file formats, cloud storage, and HTTP servers. Step - Each Step runs a Block that uses specific business logic. The output of each Step is fed into the next Step, creating a chain of transformations. Blocks - The Block defines the business logic. Blocks can: - Read and write from relational and non relational databases - Read, write, and parse data from local storage and cloud storage - Perform transformations, modify structure, add computed fields, rename fields, or remove fields - Enrich data from external sources and APIs DataYoga Runtime DataYoga provides a standalone stream processing engine, the DataYoga Runtime that validates and run Transformation Jobs. The Runtime provides: - Validation - Error handling - Metrics and observability - Credentials management The Runtime supports multiple stream processing strategies including buffering and rate limiting. It supports both async processing, multi-threading, and multi-processing to enable maximum throughput with a low footprint. Quickstart Verify that the installation completed successf","default_branch":null,"files":null,"tree":[],"storefront":"/r/datayoga-io","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/datayoga-io/datayoga/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}