{"repo":"yennanliu/NYC_Taxi_Pipeline","free":true,"listed":false,"github":"https://github.com/yennanliu/NYC_Taxi_Pipeline","clone":"git clone https://github.com/yennanliu/NYC_Taxi_Pipeline.git","description":"Stream/batch system with Hadoop, Spark on NYC taxi data | #DE","language":"Scala","stars":26,"topics":["spark","spark-streaming","spark-batch","kafka","s3","nyc-taxi-dataset","events","stream-pipeline","scala"],"license":null,"category":"data-pipelines","readme_excerpt":"NYC Taxi Pipeline INTRO Architect batch/stream data processing systems from nyc-tlc-trip-records-data, via the ETL batch process : E (extract : tlc-trip-record-data.page - S3 ) - T (transform : S3 - Spark) - L (load : Spark - Mysql) & stream process : Event - Event digest - Event storage. The system then can support calculation such as Top Driver By area , Order by time windiw , latest-top-driver , and Top busy areas . Blog Intro Batch data : nyc-tlc-trip-records-data Stream data : TaxiEvent, stream from file . Tech : Spark, Hadoop, Hive, EMR, S3, MySQL, Kinesis, DynamoDB , Scala, Python, ELK, Kafka Batch pipeline : DataLoad - DataTransform - CreateView - SaveToDB - SaveToHive Download batch data : download sample data.sh Batch data : transactional-data, reference-data - processed-data - output-transactions - output-materializedview Stream pipeline : TaxiEvent - EventLoad - KafkaEventLoad Stream data : taxi-event Please also check NYC Taxi Trip Duration in case you are interested in the data science projects with similar taxi dataset. Architecture - Architecture idea (Batch): - Architecture idea (Stream): File structure Prerequisites Prerequisites - Install (batch) - Spark 2.4.3 - Java 1.8.0 11 (java 8) - Scala 2.11.12 - sbt 1.3.5 - Mysql - Hive (optional) - Hadoop (optional) - Python 3 (optional) - Pyspark (optional) - Install (stream) - Zoopkeeper - Kafka - Elasticsearch 7.6.1 - https://www.elastic.co/downloads/elasticsearch - Kibana 7.6.1 - https://www.elastic.co/downloads","default_branch":null,"files":null,"tree":[],"storefront":"/r/yennanliu","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/yennanliu/NYC_Taxi_Pipeline/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}