{"repo":"dimajix/flowman","free":true,"listed":false,"github":"https://github.com/dimajix/flowman","clone":"git clone https://github.com/dimajix/flowman.git","description":"Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pipelines.","language":"Scala","stars":95,"topics":["hadoop","spark","scala","etl","flowman","data-engineering","sql","apache-spark","big-data","bigdata"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Flowman The declarative data build tool based on Apache Spark. 🤔 What is Flowman? Flowman is a data build tool based on Apache Spark that simplifies the act of implementing data transformation logic as part of complex data pipelines. Flowman follows a strict \"everything-as-code\" approach, where the whole transformation logic is specified in purely declarative YAML files. These describe all details of the data sources, sinks and data transformations. This is much simpler and efficient than writing Spark jobs in Scala or Python. Flowman will take care of all the technical details of a correct and robust implementation and the developers can concentrate on the data transformations themselves. In addition to writing and executing data transformations, Flowman can also be used for managing physical data models, i.e. Hive or SQL tables. Flowman can create such tables from a specification with the correct schema and also automatically perform migrations. This helps to keep all aspects (like transformations and schema information) in a single place managed by a single tool. 💪 Noteable Features Semantics of a build tool like Maven — just for data instead for applications Declarative syntax in YAML files Data model management (Create, Migrate and Destroy Hive tables, JDBC tables or file based storage) Generation of meaningful data model documentation Flexible expression language for parametrizing a project for different environments (DEV, TEST, PROD) Jobs for managing build targets (","default_branch":null,"files":null,"tree":[],"storefront":"/r/dimajix","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/dimajix/flowman/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}