{"repo":"pregismond/etl-data-pipelines-with-shell-airflow-kafka","free":true,"listed":false,"github":"https://github.com/pregismond/etl-data-pipelines-with-shell-airflow-kafka","clone":"git clone https://github.com/pregismond/etl-data-pipelines-with-shell-airflow-kafka.git","description":"Traffic Flow Optimization with Airflow and Kafka","language":"Python","stars":12,"topics":["airflow","etl-pipeline","kafka","python","shellscript","coursera","data-engineering","ibm-skills-network","bashoperator","pythonoperator"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Traffic Flow Optimization with Airflow and Kafka Disclaimer This repository contains my submission for the Final Assignment, consisting of Hands-on Lab: Build an ETL Pipeline using Bash with Airflow and Hands-on Lab: Build a Streaming ETL Pipeline using Kafka . The original files were provided by the IBM Skills Network as part of the ETL and Data Pipelines with Shell, Airflow and Kafka course on Coursera. Usage You are welcome to use this repository as a reference or starting point for your own assignment. If you choose to fork this repository, please ensure that you comply with the terms of the Apache License and give proper credit to the orginal authors. Objectives Create an ETL pipeline using an Airflow DAG Build a streaming ETL pipeline using Kafka Project Scenario As a data engineer, I am tasked with a project that seeks to alleviate congestion on national highways. To do so, I must analyze road traffic data from various toll plazas. Here’s the catch: each highway is managed by a different toll operator, and their IT systems employ different file formats. My challenge lies in harmonizing this disparate data to derive meaningful insights and contribute to smoother traffic flow. Following that, I need to create a data pipeline to collect the timestamp data streamed to Kafka and load it into a database. The high-level approach was to: Collect data available in different formats and consolidate it into a single file. Create a data pipeline that collects the streaming data an","default_branch":null,"files":null,"tree":[],"storefront":"/r/pregismond","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/pregismond/etl-data-pipelines-with-shell-airflow-kafka/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}