{"repo":"gizmodata/spark-connect-proxy","free":true,"listed":false,"github":"https://github.com/gizmodata/spark-connect-proxy","clone":"git clone https://github.com/gizmodata/spark-connect-proxy.git","description":"A reverse proxy server which allows secure connectivity to a Spark Connect server","language":"Python","stars":16,"topics":["aws","emr","spark","spark-connect","spark-connect-server","apache-spark","ibis","jwt-authentication","pyspark","tls-support"],"license":null,"category":"networking-infra","readme_excerpt":"spark-connect-proxy A reverse proxy server which allows secure connectivity to a Spark Connect server. Why? Because Spark Connect does NOT provide authentication and/or TLS encryption out of the box. This project provides a reverse proxy server which can be used to secure the connection to a Spark Connect server. Setup (to run locally) Install Python package You can install spark-connect-proxy from PyPi or from source. Option 1 - from PyPi Option 2 - from source - for development Note For the following commands - if you running from source and using --editable mode (for development purposes) - you will need to set the PYTHONPATH environment variable as follows: Usage This repo contains scripts to let you provision an AWS EMR Spark cluster with a secure Spark Connect Proxy server to allow you to securely and remotely connect to it. First - you'll need to open up a port for public access to the AWS EMR Spark Cluster - in addition to the ssh port: 22 . Add port: 50051 as shown here: [!NOTE] Even though you are opening this port to the public, the Spark Connect Proxy will secure it with TLS and JWT Authentication. The scripts use the AWS CLI to provision the EMR Spark cluster - so you will need to have the AWS CLI installed and configured with your AWS credentials. You can create a file in your local copy of the scripts directory called .env with the following contents: To provision the EMR Spark cluster - run the following command from the root directory of this repo: That will ","default_branch":null,"files":null,"tree":[],"storefront":"/r/gizmodata","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/gizmodata/spark-connect-proxy/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}