{"repo":"hussein-awala/spark-on-k8s","free":true,"listed":false,"github":"https://github.com/hussein-awala/spark-on-k8s","clone":"git clone https://github.com/hussein-awala/spark-on-k8s.git","description":"A Python package to submit and manage Apache Spark applications on Kubernetes.","language":"Python","stars":47,"topics":["kubernetes","python","spark","airflow"],"license":"Apache-2.0","category":"deployment-docker-iac","readme_excerpt":"Spark On Kubernetes Spark on Kubernetes is a python package that makes it easy to submit and manage spark apps on Kubernetes. It provides a Python client that can be used to submit apps in your API or scheduler of choice, and a CLI that can be used to submit apps from the command line, instead of using spark-submit. It also provides an optional REST API with a web UI that can be used to list and manage apps, and access the spark UI through the reverse proxy. Installation To install the core python package (only the Python client and the helpers), run: If you want to use the REST API and the web UI, you will also need to install the api package: You can also install the package from source with pip or poetry: Usage Setup the Kubernetes namespace When submitting a Spark application to Kubernetes, we only create the driver pod, which is responsible for creating and managing the executors pods. To give the driver pod the permissions to create the executors pods, we can give it a service account with the required permissions. To simplify this process, we provide a helper function that creates a namespace if needed, and a service account with the required permissions: With Python: With the CLI: Python Client The Python client can be used to submit apps from your Python code, instead of using spark-submit: CLI The CLI can be used to submit apps from the command line, instead of using spark-submit, it can also be used to manage apps submitted with the Python client (list, get, delete","default_branch":null,"files":null,"tree":[],"storefront":"/r/hussein-awala","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/hussein-awala/spark-on-k8s/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}