{"repo":"AbsaOSS/pramen","free":true,"listed":false,"github":"https://github.com/AbsaOSS/pramen","clone":"git clone https://github.com/AbsaOSS/pramen.git","description":"Resilient data pipeline framework running on Apache Spark","language":"Scala","stars":31,"topics":["hacktoberfest","big-data","data-pipeline","etl","scala","spark"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"About Pramen Pramen is a framework for defining data pipelines based on Spark and a configuration driven tool to run and coordinate those pipelines. The project focuses around Hadoop and Spark, but can run arbitrary jobs. The idea behind Pramen pipelines is simple. A pipeline consists of Sources are the data systems that are not managed by the pipeline. An example could be an operational relational database. - Ingestion jobs are used to get data from external systems into the metastore. Metastore is the data storage managed by the pipeline. Data in the metastore is accessed by table names. The metastore hides the underlying storage and format, which is usually Parquet or Delta on HDFS or S3. - Transformation jobs are used to transform data from the metastore and save the results back to the metastore. Transformers can be written in Scala or in Python. Sinks are targets to send data from the metastore. An example could be a Kafka cluster or a CSV file in a local folder. - Sink jobs are used to send data from the metastore to sinks The architecture is customizable, so you can define your own sources, transformers and sinks and deploy it independently from the framework. With Pramen you can: Build a data lake for tabular data. - Define ingestion jobs to get data from external data sources to HDFS or S3. - Organize data by partitioning it according to event or snapshot date. Create ETL data pipelines - Define ingestion jobs to extract data from external sources to the metastore. ","default_branch":null,"files":null,"tree":[],"storefront":"/r/AbsaOSS","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/AbsaOSS/pramen/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}