{"repo":"hoangsonww/End-to-End-Data-Pipeline","free":true,"listed":false,"github":"https://github.com/hoangsonww/End-to-End-Data-Pipeline","clone":"git clone https://github.com/hoangsonww/End-to-End-Data-Pipeline.git","description":"📈 A scalable, production-ready data pipeline for real-time streaming & batch processing, integrating Kafka, Spark, Airflow, AWS, Kubernetes, and MLflow. Supports end-to-end data ingestion, transformation, storage, monitoring, and AI/ML serving with CI/CD automation using Terraform & GitHub Actions.","language":"Python","stars":136,"topics":["airflow","apache","docker","elasticsearch","flink","grafana","great-expectations","hadoop","influxdb","kafka"],"license":"MIT","category":"data-pipelines","readme_excerpt":"End-to-End Data Pipeline A production-ready, fully containerized data platform with batch ingestion, real-time streaming, a star-schema data warehouse, ML experiment tracking, a .NET 8 REST API, and full observability -- all orchestrated through 20 Docker services managed by a single docker compose stack. Table of Contents - Architecture - Pipeline Flows - Technology Stack - Prerequisites - Quick Start - Service URLs - API Documentation (.NET 8 Backend) - Data Warehouse Schema - Airflow DAGs - Testing - CI/CD Pipeline - Project Structure - Configuration - Contributing - License Architecture Pipeline Flows Batch Pipeline (Daily) Streaming Pipeline (Continuous) Warehouse ETL (Hourly) Technology Stack Layer Technology Version Purpose ------- ----------- --------- --------- Orchestration Apache Airflow 2.7.3 DAG scheduling and pipeline orchestration Batch Processing Apache Spark 3.5.3 Large-scale ETL and transformations Stream Processing Apache Kafka 7.5.0 (Confluent) Event streaming and real-time ingestion Data Quality Great Expectations latest Schema validation and data quality checks Source Database MySQL 8.0 Transactional source system Data Warehouse Snowflake / PostgreSQL - / 15 Star-schema warehouse (Snowflake primary, PG fallback) Object Storage MinIO latest S3-compatible data lake Cache Redis 7-alpine Caching and session storage Document Store MongoDB 6.0.13 NoSQL storage for semi-structured data Time Series InfluxDB 2.7 IoT and time-series metrics Search Elasticsearch 8.","default_branch":null,"files":null,"tree":[],"storefront":"/r/hoangsonww","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/hoangsonww/End-to-End-Data-Pipeline/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}