{"repo":"josephmachado/beginner_de_project","free":true,"listed":false,"github":"https://github.com/josephmachado/beginner_de_project","clone":"git clone https://github.com/josephmachado/beginner_de_project.git","description":"Beginner data engineering project - batch edition","language":"HTML","stars":590,"topics":["airflow","etl","python","docker","emr","redshift","spark","redshift-cluster","database","engineering"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Beginner DE Project - Batch Edition Run Data Pipeline Run on codespaces Run locally Architecture Beginner DE Project - Batch Edition Code for blog at Data Engineering Project for Beginners. Run Data Pipeline Code available at beginner de project repository. Run on codespaces You can run this data pipeline using GitHub codespaces. Follow the instructions below. 1. Create codespaces by going to the beginner de project repository, cloning it(or click Use this template button) and then clicking on Create codespaces on main button. 2. Wait for codespaces to start, then in the terminal type make up . 3. Wait for make up to complete, and then wait for 30s (for Airflow to start). 4. After 30s go to the ports tab and click on the link exposing port 8080 to access Airflow UI (username and password is airflow ). Note Make sure to switch off codespaces instance, you only have limited free usage; see docs here. Run locally To run locally, you need: 1. git 2. Github account 3. Docker with at least 4GB of RAM and Docker Compose v1.27.0 or later Clone the repo and run the following commands to start the data pipeline: Go to http:localhost:8080 to see the Airflow UI. Username and password are both airflow . Architecture This data engineering project, includes the following: 1. Airflow : To schedule and orchestrate DAGs. 2. Postgres : To store Airflow's details (which you can see via Airflow UI) and also has a schema to represent upstream databases. 3. DuckDB : To act as our warehouse 4. Quart","default_branch":null,"files":null,"tree":[],"storefront":"/r/josephmachado","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/josephmachado/beginner_de_project/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}