{"repo":"LucaCanali/sparkMeasure","free":true,"listed":false,"github":"https://github.com/LucaCanali/sparkMeasure","clone":"git clone https://github.com/LucaCanali/sparkMeasure.git","description":"This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.","language":"Scala","stars":828,"topics":["spark","apache-spark","performance-troubleshooting","performance-metrics","python","scala"],"license":"Apache-2.0","category":"analytics","readme_excerpt":"SparkMeasure - a performance tool for Apache Spark SparkMeasure is a tool and a library designed to ease performance measurement and troubleshooting of Apache Spark jobs. It focuses on easing the collection and analysis of Spark metrics, making it a practical choice for both developers and data engineers. With sparkMeasure, users gain a deeper understanding of their Spark job performance, enabling faster and more reliable data processing workflows. ✨ Highlights - Interactive Troubleshooting: Ideal for real-time analysis of Spark workloads in notebooks and spark-shell/pyspark environments. - Development & CI/CD Integration: Facilitates testing, measuring, and comparing execution metrics of Spark jobs under various configurations or code changes. - Batch Job Analysis: With Flight Recorder mode sparkMeasure transparently records batch job metrics for later analysis. - Monitoring Capabilities: Integrates with external systems like Apache Kafka, Prometheus Push Gateway, Prometheus JMX Exporter, and InfluxDB for extensive monitoring. - Educational Tool: Serves as a practical example of implementing Spark Listeners for the collection of detailed Spark task metrics. - Language Compatibility: Fully supports Scala, Java, and Python, making it versatile for a wide range of Spark applications. 📚 Table of Contents - SparkMeasure - a performance tool for Apache Spark - ✨ Highlights - 📚 Table of Contents - Links to related work on Spark Performance - 🚀 Quick start - Examples of sparkMeas","default_branch":null,"files":null,"tree":[],"storefront":"/r/LucaCanali","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/LucaCanali/sparkMeasure/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}