{"repo":"cerndb/SparkPlugins","free":true,"listed":false,"github":"https://github.com/cerndb/SparkPlugins","clone":"git clone https://github.com/cerndb/SparkPlugins.git","description":"Code and examples of how to write and deploy Apache Spark Plugins. Spark plugins allow runnig custom code on the executors as they are initialized. This also allows extending the Spark metrics systems with user-provided monitoring probes.","language":"Scala","stars":96,"topics":["kubernetes","monitoring","scala","spark"],"license":"Apache-2.0","category":"analytics","readme_excerpt":"SparkPlugins Spark Plugins are an Apache Spark feature for extending Spark with custom metrics and actions. This repository provides ready-to-use examples for deploying Spark plugins across various use cases. Key Features Custom Metrics: Extend Spark's instrumentation with user-defined metrics. Executor Actions: Trigger custom actions upon executor startup, useful for integrations (e.g., monitoring systems). Resource Monitoring: Measure Spark’s usage of cluster resources (YARN, K8S, Standalone). I/O Metrics: Monitor I/O performance from cloud filesystems, OS metrics, and custom application metrics. External Integrations: Connect with external systems like Pyroscope for performance insights. Contents - Getting started - Demo and basic plugins - Implementation notes - Plugin for integrating Pyroscope with Spark - Plugin for OS metrics instrumentation with Cgroups for Spark on Kubernetes - Plugin to collect I/O storage statistics for HDFS and Hadoop-compatible filesystems - Plugin for Cloud filesystem storage statistics - Experimental plugins Resources - Spark Performance Dashboard - a solution to ingest and visualize Spark metrics - link to the repo on how to deploy a Spark Performance Dashboard using Spark metrics - DATA+AI summit 2020 talk What is New with Apache Spark Performance Monitoring in Spark 3.0 - DATA+AI summit 2021 talk Monitor Apache Spark 3 on Kubernetes using Metrics and Plugins Author and contact: Luca.Canali@cern.ch --- Getting Started - Your First Spark Plugi","default_branch":null,"files":null,"tree":[],"storefront":"/r/cerndb","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/cerndb/SparkPlugins/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}