{"repo":"iobruno/data-engineering-labs","free":true,"listed":false,"github":"https://github.com/iobruno/data-engineering-labs","clone":"git clone https://github.com/iobruno/data-engineering-labs.git","description":"Data Engineering examples for Airflow, Prefect; dbt for BigQuery, Redshift, ClickHouse, Postgres, DuckDB; PySpark for Batch processing; Kafka for Stream processing","language":"Python","stars":79,"topics":["pyspark","spark","kafka","ksqldb","dbt-bigquery","dbt-postgres","dbt-clickhouse","dbt-redshift","airflow","airflow-dags"],"license":"CC-BY-SA-4.0","category":"data-pipelines","readme_excerpt":"Data Engineering Labs A hands-on collection of data engineering projects. The module structure is inspired by the Data Engineering Zoomcamp from DataTalks.Club. All implementations are original work. Refer to the modules below for covered topics and tools. Modules Module 1: Data ingestion Python ingestion with polars and pandas Rust data ingestion ELT Ingestion with Airbyte data load tool (dlt) IaC with Terraform (Google Cloud Platform) Module 2: Workflow orchestration Workflow orchestration with Airflow 3.x Workflow orchestration with Airflow 2.x Workflow orchestration with Prefect Module 3: Lakehouses & Data Warehouse BigQuery Data Warehouse StarRocks Query Engine Module 4: Analytics engineering BigQuery and dbt Snowflake and dbt Databricks and dbt Redshift and dbt ClickHouse and dbt PostgreSQL and dbt DuckDB and dbt Data visualization with Metabase Data visualization with Superset Module 5: Batch processing PySpark 4.x + Spark Connect PySpark 3.x + Spark Connnect Spark (Scala) Module 6: Stream processing PyFlink Stream processing with Kafka, ksqlDB and Kotlin Kafka Streams with ksqlDB Extras LakeHouse with Delta, Iceberg, Hive Data Catalog with DataHub","default_branch":null,"files":null,"tree":[],"storefront":"/r/iobruno","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/iobruno/data-engineering-labs/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}