{"repo":"HoracioSoldman/batch-processing-on-aws","free":true,"listed":false,"github":"https://github.com/HoracioSoldman/batch-processing-on-aws","clone":"git clone https://github.com/HoracioSoldman/batch-processing-on-aws.git","description":"With everything I learned from DEZoomcamp from datatalks.club, this project performs a batch processing on AWS for the cycling dataset which is available on Transport For London (TFL) website. https://cycling.data.tfl.gov.uk","language":"Jupyter Notebook","stars":15,"topics":["airflow","aws","data-engineering","etl","spark"],"license":null,"category":"data-pipelines","readme_excerpt":"Batch processing on aws This project shows one way to perform a batch processing using mainly AWS and a few open-source tools. Table of contents - Overview - The Goal - The dataset - Data modeling - Tools - Scalability - Running the project 1. Requirements 2. Clone the repository 3. Run Terraform 4. Create the Data Warehouse 5. Run Airflow 6. Run the Airflow DAGs 7. Visualise data on Metabase - Project limitations Manual DAGs triggering Overview The current work aims to give answers to business questions concerning bicycle rentals in the city of London from 2021 to January 2022. To do so, we are going to build a data pipeline which collects data from multiple sources, applies transformations and displays the preprocessed data into a dashboard. The following diagram illustrates a high-level structure of the pipeline where data flows from different sources to the final visualisation tool. The Goal The end goal of the current project is to preprocess the data on AWS platform and get useful insights from it. We can learn more from the data by responding to some of the following business questions on the final dashboard. - At what time or which hour of the day has the most active rental in average? - Which area has the most active bike rentals in London? - Which day of the week is the most active in general? - What is the global trend for daily rentals over the year? The dataset We are going to process 3 datasets along this project. 1. Cycling journey dataset from January 2021 to ","default_branch":null,"files":null,"tree":[],"storefront":"/r/HoracioSoldman","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/HoracioSoldman/batch-processing-on-aws/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}