{"repo":"chayansraj/Python-ETL-pipeline-using-Airflow-on-AWS","free":true,"listed":false,"github":"https://github.com/chayansraj/Python-ETL-pipeline-using-Airflow-on-AWS","clone":"git clone https://github.com/chayansraj/Python-ETL-pipeline-using-Airflow-on-AWS.git","description":"This project demonstrates how to build and automate an ETL pipeline written in Python and schedule it using open source Apache Airflow orchestration tool on AWS EC2 instance.","language":"Python","stars":24,"topics":["airflow","airflow-dag","automation","aws","aws-ec2","dags","data-engineering","etl-pipeline","python3","ssh-server"],"license":null,"category":"data-pipelines","readme_excerpt":"Python ETL pipeline using Airflow on AWS This project demonstrates how to build and automate an ETL pipeline written in Python and schedule it using open source Apache Airflow orchestration tool on AWS EC2 instance. Project Goals 1. Data Ingestion - Create a data ingestion pipeline to extract data from OpenWeather API. 2. Data Storage - Create a data storage repository using AWS S3 buckets. 3. Data Transformation - Create ETL job to extract the data, do simple transformations and load the clean data using Airflow. 4. Data Pipeline - Create a data pipeline written in Python that extracts data from API calls and store it in AWS S3 buckets. 5. Pipeline Automation - Create scheduling service using Apace Airflow to trigger the data pipeline and automate the process. Data Architecture The architecture (Data flow) used in this project uses different Open source and cloud components as described below: Source: Author Dataset Used In this project, we are building a data pipeline that extracts data through an API call to Openweathermap. API calls are made using Airflow through HTTPoperator. The API call looks something like this : where there are two parameters 1. q - City name: Takes the name of any city in the world, the name goes in lowercase 2. API key - It can be obtained after creating an account on Openweathermap.org. The API key is unique to each account, make sure you do not share it. The resulting data is in json format with the following structure with an example call of Lon","default_branch":null,"files":null,"tree":[],"storefront":"/r/chayansraj","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/chayansraj/Python-ETL-pipeline-using-Airflow-on-AWS/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}