{"repo":"GoogleCloudPlatform/dataproc-scala-examples","free":true,"listed":false,"github":"https://github.com/GoogleCloudPlatform/dataproc-scala-examples","clone":"git clone https://github.com/GoogleCloudPlatform/dataproc-scala-examples.git","description":"Dataproc Scala Examples is an effort to assist in the creation of Spark jobs written in Scala to run on Dataproc.","language":"Scala","stars":12,"topics":["airflow","composer","dataproc","gcp","scala","spark"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Dataproc Scala Examples Dataproc Scala Examples is an effort to assist in the creation of Spark jobs written in Scala to run on Dataproc. Google is providing different pre-implemented Spark jobs and technical guides to run them on GCP. This guide is based on the WordCount ETL example with common sources and sinks (Kafka, GCS, BigQuery, etc). It is intended to catalyze your development to run Spark jobs written in Scala on Dataproc. It is demonstrated how to run Spark jobs using Dataproc Submit, Serverless, Workflow and how to orchestrate them with Cloud Composer. Templates If you are looking to use Dataproc Templates, please refer to this repository. Quickstarts Check out the quickstart documentation for quickstarts. Dataproc Scala Examples - Versions Note: if using Dataproc Serverless (detailed in the guides as one of the options to run jobs), please recompile the jobs using Spark version 3.2.0 Before you start - Be aware that the data format used in this guide for data in GCS is Parquet. - This guide is configured to run the main class, despite Dataproc having the option to specify another class to run. Setup your Scala environment Follow the setup instructions for installing, testing and compiling the project. Use the Cloud Shell environment in GCP Dataproc Spark Use Cases 0) Create Mock Dataset - Creates input and output mock WordCount datasets in GCS and BQ to use in other examples 1) Streaming - Kafka to GCS - Runs a Spark Structured Streaming WordCount example from Kaf","default_branch":null,"files":null,"tree":[],"storefront":"/r/GoogleCloudPlatform","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/GoogleCloudPlatform/dataproc-scala-examples/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}