{"repo":"nsphung/pyspark-template","free":true,"listed":false,"github":"https://github.com/nsphung/pyspark-template","clone":"git clone https://github.com/nsphung/pyspark-template.git","description":"A Python PySpark Projet with Poetry","language":"Jupyter Notebook","stars":31,"topics":["poetry","pyspark","project","python3","template","black","pytest","isort","jupyter-notebook","spark"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"PySpark Template Project # People has asked me several times how to setup a good/clean/code organization for Python project with PySpark. I didn't find a fully feature project, so this is my attempt for one. Moreover, have a simple integration with Jupyter Notebook inside the project too. Deepwiki Documentation Deepwiki: https://deepwiki.com/nsphung/pyspark-template Table of Contents - PySpark Template Project - Usage - Inspiration - Development - Prerequisites - [Pyenv Manual Install \\[Optional\\]](#pyenv-manual-install-optional) - Add format, lint code tools - Autolint/Format code with Black in IDE: - Checked optional type with Mypy PEP 484 - Isort - Fix - Usage Local - Use with poetry - Usage in distributed-mode depending on your cluster manager type Usage Inspiration https://mungingdata.com/pyspark/chaining-dataframe-transformations/ https://medium.com/albert-franzi/the-spark-job-pattern-862bc518632a https://pawamoy.github.io/copier-poetry/ https://drivendata.github.io/cookiecutter-data-science/#why-use-this-project-structure Development Prerequisites All you need is the following configuration already installed: Git The project was tested with Python 3.10.18 managed by pyenv: Use make pyenv goal to launch the automated install of pyenv JAVA HOME environment variable configured with a Java JDK11 SPARK HOME environment variable configured with Spark version spark-3.5.6-bin-hadoop3 package PYSPARK PYTHON environment variable configured with \"python3.10\" PYSPARK DRIVER PYTHON","default_branch":null,"files":null,"tree":[],"storefront":"/r/nsphung","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/nsphung/pyspark-template/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}