{"repo":"dain55788/ELT-Data-Pipeline","free":true,"listed":false,"github":"https://github.com/dain55788/ELT-Data-Pipeline","clone":"git clone https://github.com/dain55788/ELT-Data-Pipeline.git","description":"ELT Data Pipeline implementation in Data Warehousing environment","language":"Jupyter Notebook","stars":31,"topics":["apache-airflow","data-engineering","dbt","great-expectations","postgresql","powerbi","apache-spark","minio","trino"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"ELT-Data-Pipeline 🚀 ELT Data Pipeline Project implementation in Data Warehousing environment using DBT, Airflow, PostgreSQL and more!! ⚡ --- 📕 What is covered in this project? + Data Orchestration : Apache Airflow + Data Processing : Apache Spark + On-Premise Data Lake : MinIO (Object Storage), Hive Metastore (Metadata Layer) + Query Engine : Trino, DBeaver + Data Warehousing : PostgreSQL + Data Governance and Data Quality (Staging Area) : Great Expectations + Data Transformaton and Data Modeling (Star Schema) : DBT (Data Build Tool) 🌟 + Data Visualization : PowerBI 📊 + ELT Data Processing Terminology Implemention ⛅ API Data Source In this project, we will take a look at OpenAQ API, read the document here: https://openaq.org/. Specifically, we will explore the location measurement of air quality of different locations/ countries in the world. Take a look on how to create your API Key: https://docs.openaq.org/using-the-api/quick-start 🛠️ System Architecture 📁 Repository Structure ⚙ Workflow Diagram 🔥 NOTE The Project is still in progress, so stay tuned!! 🙌🏻","default_branch":null,"files":null,"tree":[],"storefront":"/r/dain55788","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/dain55788/ELT-Data-Pipeline/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}