{"repo":"StabRise/ScaleDP","free":true,"listed":false,"github":"https://github.com/StabRise/ScaleDP","clone":"git clone https://github.com/StabRise/ScaleDP.git","description":"ScaleDP is an Open-Source extension of Apache Spark for Document Processing","language":"Python","stars":19,"topics":["pdf","pdf-document-processor","spark","ocr","ocr-python","ocr-recognition","easyocr","huggingface-models","machine-learning","nlp"],"license":"AGPL-3.0","category":"media-processing","readme_excerpt":"An Open-Source Library for Processing Documents using AI/ML in Apache Spark. --- Source Code : https://github.com/StabRise/ScaleDP Quickstart : 1.QuickStart.ipynb Documentations : https://scaledp.stabrise.com Tutorials : https://github.com/StabRise/ScaleDP-Tutorials Change Log --- Welcome to the ScaleDP library ScaleDP is library allows you to process documents using AI/ML capabilities and scale it using Apache Spark. LLM (Large Language Models) and VLM (Vision Language Models) models are used to extract data from text and images in combination with OCR engines. Discover pre-trained models for your projects or play with the thousands of models hosted on the Hugging Face Hub. Key features Document processing: - ✅ Loading PDF documents/Images to the Spark DataFrame (using Spark PDF Datasource and as binaryFile ) - ✅ Extraction text/images from PDF documents/Images - ✅ Zero-Shot extraction structured data from text/images using LLM and ML models - ✅ Possibility run as REST API service without Spark Session for have minimum processing latency - ✅ Support Streaming mode for processing documents in real-time LLM: Support OpenAI compatible API for call LLM/VLM models (GPT, Gemini, GROQ, etc.) - OCR Images/PDF documents using Vision LLM models - Extract data from the image using Vision LLM models - Extract data from the text/images using LLM models - Extract data using DSPy framework - NER using LLM's - Visualize results NLP: - Extract data from the text/images using NLP models from ","default_branch":null,"files":null,"tree":[],"storefront":"/r/StabRise","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/StabRise/ScaleDP/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}