{"repo":"martinprobson/Spark-Scala-Maven-Example","free":true,"listed":false,"github":"https://github.com/martinprobson/Spark-Scala-Maven-Example","clone":"git clone https://github.com/martinprobson/Spark-Scala-Maven-Example.git","description":"Example Maven configuration for a Spark, Scala project","language":"Scala","stars":54,"topics":["apache-spark","scala","maven"],"license":"MIT","category":"api-integrations-sdks","readme_excerpt":"Example Maven Project for Scala Spark 2 Application =================================================== Introduction This archive contains an example Maven project for Scala Spark 3.x application. Details The pom.xml contains example dependencies for : - Spark SLF4J LOG4J (acts as logging implementation for SLF4J) grizzled-slf4 a Scala specific wrapper for SLF4J. typesafe for config. scalatest for testing. Note that Scala itself is just listed as another dependency which means a global installation of Scala is not required. The pom.xml builds an uber-jar containing all the dependencies by default (including Scala jars). The pom also includes two exec goals: - exec:exec@run-local - run the code using local spark instance. exec:exec@run-yarn - run the code on a remote yarn cluster. In order for this to work the hive-site.xml , core-site.xml and yarn-site.xml configuration files from the remote cluster must be copied into the spark-remote/conf directory. Note To run this code from your IDE, the following exports must be added to the JVM args:- This is also configured in the scalatest maven plugin in the pom: - Reference Maven for scala link","default_branch":null,"files":null,"tree":[],"storefront":"/r/martinprobson","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/martinprobson/Spark-Scala-Maven-Example/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}