{"repo":"swoop-inc/spark-alchemy","free":true,"listed":false,"github":"https://github.com/swoop-inc/spark-alchemy","clone":"git clone https://github.com/swoop-inc/spark-alchemy.git","description":"Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive","language":"Scala","stars":191,"topics":["spark","data-science","scala","data-engineering"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"spark-alchemy Spark Alchemy is a collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive in our demanding petabyte-scale environment with rich data (thousands of columns). Supported languages While spark-alchemy , like Spark itself, is written in Scala, much of its functionality, such as interoperable HyperLogLog functions, can be used from other Spark-supported languages such as SparkSQL and Python. Installation Add the following to your libraryDependencies in SBT: You can find all released versions here. Some use cases such as interoperability with PySpark may require the assembly of a fat JAR of spark-alchemy . To assemble, run sbt assembly . To skip tests during assembly, run sbt 'set sbt.Keys.test in assembly := {}' assembly instead. For Spark users - Native HyperLogLog functions that offer reaggregatable fast approximate distinct counting capabilities far beyond those in OSS Spark with interoperability to Postgres and even JavaScript. Just as Spark's own native functions, once the functions are registered with Spark, they can be used from SparkSQL, Python, etc. For Spark framework developers - Helpers for native function registration - Look at SparkSessionSpec as an example of how to reuse advanced Spark testing functionality from OSS Spark. For Python developers - See HyperLogLog functions for an example of how spark-alchemy HLL functions can be registered for use through PySpark. What we ","default_branch":null,"files":null,"tree":[],"storefront":"/r/swoop-inc","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/swoop-inc/spark-alchemy/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}