{"repo":"godatadriven/spot","free":true,"listed":false,"github":"https://github.com/godatadriven/spot","clone":"git clone https://github.com/godatadriven/spot.git","description":"Spark + OpenTelemetry","language":"Scala","stars":15,"topics":["apache-spark","opentelemetry","observability"],"license":"Apache-2.0","category":"analytics","readme_excerpt":"Spot: Spark-OpenTelemetry This package connects [Apache Spark™][sp-home] to [OpenTelemetry][ot-home] Tracing. This allows reporting tracing from any Spark or PySpark job to [OpenTelemetry Collector][ot-col], or directly to any [supported backend][ot-export]. Status ℹ️This project is in early development. It can be used for proof-of-concept, but the instrumentation is not comprehensive. Screenshot Here's what a simple Spark job managed by Airflow might look like, if both Airflow and Spark report traces to Grafana Tempo: Usage The recommended way to use Spot relies on [OpenTelemetry Autoconfigure][ot-auto] to obtain the OpenTelemetry configuration. You pass the spot-complete jar to spark-submit to make Spot available to your job, and configure spark.extraListeners to enable it. The jar filename embeds the two-digit Spark version, and the scala version: The spark and scala versions must match your spark process. Context Propagation To use context propagation, provide the necessary headers as SparkConf values. The default configuration uses [ traceparent ][traceparent]: All SparkConf values that start with spark.com.xebia.data.spot. are made available to the ContextPropagator . If you use a different propagator than the default, you can prefix its required keys accordingly. See also Airflow Context Propagation below. Prerequisites Instrumenting for telemetry is useless until you publish the recorded data somewhere. This might be the native metrics suite of your chosen cloud provi","default_branch":null,"files":null,"tree":[],"storefront":"/r/godatadriven","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/godatadriven/spot/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}