{"repo":"mispecto/spark-flamegraph","free":true,"listed":false,"github":"https://github.com/mispecto/spark-flamegraph","clone":"git clone https://github.com/mispecto/spark-flamegraph.git","description":"Easy CPU Profiling for Apache Spark applications","language":"Shell","stars":49,"topics":["apache-spark","flamegraph","spark","cpu-profiling"],"license":"Apache-2.0","category":"api-integrations-sdks","readme_excerpt":"spark-flamegraph ================ Easy CPU Profiling for Apache Spark applications. The script spark-submit-flamegraph is a wrapper around standard spark-submit that generates Flame Graph. Supported Systems Amazon EMR Most Linux distributions Mac (with Homebrew installed) Prerequisites The script is adapted for work in Amazon EMR. Otherwise the following utilities must present on your system: perl python2.7 (or set PYTHON environment variable to the Python executabl) pip (or set PIP environment variable to the pip utility) Running Use spark-submit-flamegraph as a replacement for the spark-submit command. Configuration To configure use the following environment variables: Environment Variable Description Default value -------------------- ------------ ------------- SPARK CMD Spark command to run spark-submit PYTHON Path to the Python executable python2.7 PIP Path to the pip utility pip For example, to profile Spark shell session set SPARK CMD environment variable: Details The script does the following operations to make profiling Spark applications as easy as possible: Downloads InfluxDB, and starts it on some random port. Starts Spark application using original spark-submit command, with the StatsD profiler Jar in its classpath and with the configuration that tells it to report statistics back to the InfluxDB instance. After running Spark application, queries all the reported metrics from the InfluxDB instance. Run a script that generates the .SVG file. Stops the InfluxDB ins","default_branch":null,"files":null,"tree":[],"storefront":"/r/mispecto","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mispecto/spark-flamegraph/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}