{"repo":"josephmachado/data_engineering_for_beginners_code","free":true,"listed":false,"github":"https://github.com/josephmachado/data_engineering_for_beginners_code","clone":"git clone https://github.com/josephmachado/data_engineering_for_beginners_code.git","description":"Code for DE101 book at https://de101.startdataengineering.com/","language":"HTML","stars":124,"topics":["airflow","dbt","python","spark","sql"],"license":null,"category":"data-pipelines","readme_excerpt":"Data Engineering for Beginners The code for SQL, Python, and data model sections are written using Spark SQL. To run the code, you will need the prerequisites listed below. Setup Prerequisites 1. git version = 2.37.1 2. Docker version = 20.10.17 and Docker compose v2 version = v2.10.2. Windows users : please setup WSL and a local Ubuntu Virtual machine following the instructions here . Install the above prerequisites on your ubuntu terminal; if you have trouble installing docker, follow the steps here (only Step 1 is necessary). Fork this repository data engineering for beginners code . After forking, clone the repo to your local machine and start the containers as shown below: Open Jupyter Lab at http://localhost:8888 and run the code at ./notebooks/starter-notebook.ipynb to create the data and check that your setup worked. After the data is created open the Airflow UI with http://localhost:8080/ and trigger the DAG and ensure that it runs successfully. Shut down After you are done, shut down the containers with","default_branch":null,"files":null,"tree":[],"storefront":"/r/josephmachado","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/josephmachado/data_engineering_for_beginners_code/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}