{"repo":"kaoutaar/end-to-end-etl-pipeline-jcdecaux-API","free":true,"listed":false,"github":"https://github.com/kaoutaar/end-to-end-etl-pipeline-jcdecaux-API","clone":"git clone https://github.com/kaoutaar/end-to-end-etl-pipeline-jcdecaux-API.git","description":"velib-v2: An ETL pipeline that employs batch and streaming jobs using Spark, Kafka, Airflow, and other tools, all orchestrated with Docker Compose.","language":"Python","stars":21,"topics":["airflow","data-engineering","etl","etl-pipeline","kafka","spark","spark-connect","spark-streaming","streamlit"],"license":null,"category":"data-pipelines","readme_excerpt":"--- End-to-end ETL pipeline - jcdecaux API The architecture consists of 2 main pipelines: Batch pipeline: after data being served to kafka, we use spark analytics engine to transform and process data in batches and send tables into a datawarehouse Stream pipeline: we use spark streaming to fetch and filter data that will be served in our web application in realtime. Jcdecaux API: The API provides information about the location of the bike stations in Europe, the availability of bikes and parking spaces in real time, in addition to other details. A single call of this API returns the most recent information about all the existing stations, \"the most recent\" could be the last update got 1–2 min ago, which doesn't really mean real-time data, we can do nothing to improve it, this is how the API works. And to make the API act like a stream source but also avoid to overload the server, a script is scheduled in Airflow to fetch data every 30 seconds, this data is then sent to kafka cluster using kafka produder. kafka: kafka receives data and store it in a topic called \"velib data\" waiting to be polled, two consumer groups are configured to consume data from kafka broker in parallel: 1) batch-consumer: using spark-connect, a spark client script is in charge of polling data from kafka in batches, is takes care of transforming and creating tables that are sent to the datawarehouse, Airflow is configured to run this job every day at 5am, 2) stream-consumer: spark structured streaming is","default_branch":null,"files":null,"tree":[],"storefront":"/r/kaoutaar","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/kaoutaar/end-to-end-etl-pipeline-jcdecaux-API/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}