{"repo":"qwshen/spark-etl-framework","free":true,"listed":false,"github":"https://github.com/qwshen/spark-etl-framework","clone":"git clone https://github.com/qwshen/spark-etl-framework.git","description":"A generic ETL framework with Spark_SQL for transforming data by constructing pipelines with Yaml/Json/Xml","language":"Scala","stars":21,"topics":["apache-spark","etl","framwork","json","spark","spark-sql","sql","sql-query","xml","yaml"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"The Spark-etl-framework is a pipeline-based data transformation framework using Spark-SQL. For one process flow to transform and move data from end to end, a pipeline is defined. At the start of a pipeline, read-actors (readers) are required to load data from the source(s); in the middle of the pipeline, data normally gets transformed with Spark-SQL based transformers; and finally, at the end of the pipeline, write-actors (writers) write outputs to the target location. A pipeline is not limited to the order of read-transform-write, any of these 3 actions can be in any stages of a pipeline except write cannot be the first action. Also, custom actors can be developed and plugged-in. Technically a pipeline consists of multiple jobs, and each job contains multiple actions which are represented by Actors. Each job can run under the same Spark Session or separate Spark Sub-Sessions; and dataframes (as views) can be shared across jobs. Most actors require input view(s) and produce output view(s). To build the project, run Pipeline definition The following explains the definition of each section in a pipeline: 1. Settings - singleSparkSession - when a pipeline consists of multiple jobs, each job can be executed under a separate Spark Sub-Session. This provides resource isolation across jobs. When this flag is set to true, all jobs are executed under the same global Spark-Session. The default value is false. - globalViewAsLocal - global dataframes (views) are shared across jobs (even ","default_branch":null,"files":null,"tree":[],"storefront":"/r/qwshen","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/qwshen/spark-etl-framework/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}