{"repo":"prophecy-io/spark-ai","free":true,"listed":false,"github":"https://github.com/prophecy-io/spark-ai","clone":"git clone https://github.com/prophecy-io/spark-ai.git","description":"Toolbox for building Generative AI applications on top of Apache Spark.","language":"Python","stars":51,"topics":["data-engineering","generative-ai","machine-learning","pipeline","spark","spark-streaming"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Spark AI Toolbox for building Generative AI applications on top of Apache Spark. Many developers are companies are trying to leverage LLMs to enhance their existing applications or build completely new ones. Thanks to LLMs most of them no longer have to train new ML models. However, still the major challenge is data and infrastructure. This includes data ingestion, transformation, vectorization, lookup, and model serving. Over the last few months, the industry has seen a spur of new tools and frameworks to help with these challenges. However, none of them are easy to use, deploy to production, nor can deal with the scale of data. This project aims to provide a toolbox of Spark extensions, data sources, and utilities to make building robust data infrastructure on Spark for Generative AI applications easy. Example Applications Complete examples that anyone can start from to build their own Generative AI applications. - Chatbot Template - Medical Advisor Template Read about our thoughts on Prompt engineering, LLMs, and Low-code here. Quickstart Installation Currently, the project is aimed mainly at PySpark users, however, because it also features high-performance connectors, both the PySpark and Scala dependencies have to be present on the Spark cluster. Ingestion Pre-processing & Vectorization Inference Roadmap Data sources supported: - 🚧 Slack - 🗺️ PDFs - 🗺️ Asana - 🗺️ Notion - 🗺️ Google Drive - 🗺 Web-scrape Vector databases supported: - 🚧 Pinecone - 🚧 Spark-ML (table ","default_branch":null,"files":null,"tree":[],"storefront":"/r/prophecy-io","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/prophecy-io/spark-ai/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}