{"repo":"drmingler/smart-llm-loader","free":true,"listed":false,"github":"https://github.com/drmingler/smart-llm-loader","clone":"git clone https://github.com/drmingler/smart-llm-loader.git","description":"smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. Spend less time on preprocessing headaches and more time building what matters. From RAG systems to chatbots to document Q&A, SmartLLMLoader handles the heavy lifting so you can focus on creating exceptional AI applications.","language":"Python","stars":328,"topics":["chatbot","chunking","claude","gemini","langchain","llama-index","markdown","openai","pdf-converter","pdf-parser"],"license":"MIT","category":"ai-agents","readme_excerpt":"SmartLLMLoader smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. It handles the entire document processing pipeline: - 📄 Converts documents to clean markdown - 🔍 Built-in OCR for scanned documents and images - ✂️ Smart, context-aware text chunking - 🔌 Seamless integration with LangChain and LlamaIndex - 📦 Ready for vector stores and LLM ingestion Spend less time on preprocessing headaches and more time building what matters. From RAG systems to chatbots to document Q&A, SmartLLMLoader handles the heavy lifting so you can focus on creating exceptional AI applications. SmartLLMLoader's chunking approach has been benchmarked against traditional methods, showing superior performance particularly when paired with Google's Gemini Flash model. This combination offers an efficient and cost-effective solution for document chunking in RAG systems. View the detailed performance comparison here. Features - Support for multiple LLM providers - In-built OCR for scanned documents and images - Flexible document type support - Supports different chunking strategies such as: context-aware chunking and page-based chunking - Supports custom prompts and custom chunking Installation System Dependencies First, install Poppler if you don't have it already (required for PDF processing): Ubuntu/Debian: macOS: Windows: 1. Download the latest Poppler for Windows 2. Extract the downloaded file 3. Add the bin directory to your system PATH","default_branch":null,"files":null,"tree":[],"storefront":"/r/drmingler","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/drmingler/smart-llm-loader/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}