{"repo":"SETL-Framework/setl","free":true,"listed":false,"github":"https://github.com/SETL-Framework/setl","clone":"git clone https://github.com/SETL-Framework/setl.git","description":"A simple Spark-powered ETL framework that just works 🍺","language":"Scala","stars":186,"topics":["spark","etl","framework","scala","setl","pipeline","data-transformation","data-science","data-engineering","data-analysis"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"---------- If you’re a data scientist or data engineer , this might sound familiar while working on an ETL project: - Switching between multiple projects is a hassle - Debugging others’ code is a nightmare - Spending a lot of time solving non-business-related issues SETL (pronounced \"settle\") is a Scala ETL framework powered by Apache Spark that helps you structure your Spark ETL projects, modularize your data transformation logic and speed up your development. Use SETL In a new project You can start working by cloning this template project. In an existing project To use the SNAPSHOT version, add Sonatype snapshot repository to your pom.xml Quick Start Basic concept With SETL, an ETL application could be represented by a Pipeline . A Pipeline contains multiple Stages . In each stage, we could find one or several Factories . The class Factory[T] is an abstraction of a data transformation that will produce an object of type T . It has 4 methods ( read , process , write and get ) that should be implemented by the developer. The class SparkRepository[T] is a data access layer abstraction. It could be used to read/write a Dataset[T] from/to a datastore. It should be defined in a configuration file. You can have as many SparkRepositories as you want. The entry point of a SETL project is the object io.github.setl.Setl , which will handle the pipeline and spark repository instantiation. Show me some code You can find the following tutorial code in the starter template of SETL. Go and","default_branch":null,"files":null,"tree":[],"storefront":"/r/SETL-Framework","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/SETL-Framework/setl/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}