{"repo":"guidok91/spark-movies-etl","free":true,"listed":false,"github":"https://github.com/guidok91/spark-movies-etl","clone":"git clone https://github.com/guidok91/spark-movies-etl.git","description":"Spark data pipeline that processes movie ratings data.","language":"Python","stars":31,"topics":["spark","pyspark","etl","data-engineering","elt","data-pipeline","uv","apache-iceberg"],"license":null,"category":"data-pipelines","readme_excerpt":"Movies data ETL (Spark) Spark data pipeline that processes movie ratings data. - Data Architecture - Data pipeline design - Packaging and dependency management - CI/CD - Local development/execution instructions Data Architecture We define a Data Lakehouse architecture with the following layers: - Raw : Contains raw data directly ingested from an event stream, e.g. Kafka. This data should generally not be shared (can contain PII, duplicates, quality issues, etc). - Curated : Contains transformed data according to business and data quality rules. This data can be shared and accessed as tables registered in a data catalog. Apache Iceberg is used as the table format for both the raw and curated layers. Data pipeline design The Spark data pipeline: - Reads data from the raw layer ( movie ratings raw table) incrementally for a given date (filtering by ingestion date ). - Performs data cleaning, transformations and business logic. - Writes to the curated layer ( movie ratings curated table) partitioned by days(timestamp) (leveraging Iceberg's hidden partitioning for optimal querying). - After persisting, Data Quality checks are run on the curated data using Pandera. The data is processed using the Write-Audit-Publish pattern (using Iceberg's branching): data is written to an audit branch, and only after the data quality checks pass, the changes are visible in the main branch. Note that for the purpose of running this project locally, we use an Iceberg catalog in the local file syste","default_branch":null,"files":null,"tree":[],"storefront":"/r/guidok91","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/guidok91/spark-movies-etl/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}