{"repo":"abeltavares/batch-data-pipeline","free":true,"listed":false,"github":"https://github.com/abeltavares/batch-data-pipeline","clone":"git clone https://github.com/abeltavares/batch-data-pipeline.git","description":"🦆 Batch data pipeline with Airflow, DuckDB, Delta Lake, Trino, MinIO, and Metabase. Full observability and data quality.","language":"Python","stars":90,"topics":["airflow","data-lineage","delta-lake","docker","duckdb","etl","grafana","lakehouse","metabase","promotheus"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Batch Data Pipeline with Airflow, DuckDB, Delta Lake, Trino and Metabase --- 📖 Overview This project demonstrates a batch data pipeline following the Medallion Architecture (Bronze → Silver → Gold). It showcases how to ingest, clean, validate, aggregate, and visualize sales data using modern data engineering tools, all containerized with Docker for easy deployment. 🔎 Technologies Used: - Orchestration: Apache Airflow - Data Processing: DuckDB - Data Lake Format: Delta Lake - Object Storage: MinIO (S3-compatible) - Query Engine: Trino - Visualization: Metabase - Data Quality: Soda Core - Lineage: OpenLineage + Marquez - Observability: Prometheus + Grafana This pipeline is perfect for data engineers and students interested in learning how to design batch data systems with data quality, observability, and lineage tracking. --- 🏗️ Architecture Data Flow: 1. Data Generator simulates realistic sales transactions with intentional data quality issues ( 20% dirty data) 2. Bronze Layer ingests raw JSON data to MinIO without validation 3. Silver Layer cleans, validates, and deduplicates data into Delta Lake tables 4. Quality Checks validates Silver data using Soda Core with 15+ checks 5. Gold Layer creates business aggregations (daily sales, product performance, customer segments) 6. Trino provides SQL interface for querying Delta Lake tables 7. Metabase visualizes business metrics through interactive dashboards 8. Marquez tracks data lineage across the entire pipeline 9. Grafana mon","default_branch":null,"files":null,"tree":[],"storefront":"/r/abeltavares","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/abeltavares/batch-data-pipeline/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}