{"repo":"ALucek/QuicKB","free":true,"listed":false,"github":"https://github.com/ALucek/QuicKB","clone":"git clone https://github.com/ALucek/QuicKB.git","description":"Optimize Document Retrieval with Fine-Tuned KnowledgeBases","language":"Python","stars":186,"topics":["embedding-models","fine-tuning","knowledge-base","retrieval-augmented-generation","sentence-transformers","synthetic-dataset-generation"],"license":"MIT","category":"productivity","readme_excerpt":"QuicKB Optimize Document Retrieval with Fine-Tuned KnowledgeBases Overview QuicKB optimizes document retrieval by creating fine-tuned knowledgebases through an end-to-end machine learning pipeline that handles document chunking, synthetic data generation, and embedding model training. Key Features Document Chunking Implement state-of-the-art chunking strategies based on ChromaDB's research: - Semantic Approaches : - LLM-guided splits for natural language breakpoints - Content-aware clustering for thematic coherence - Hybrid semantic-token methods for balanced chunking - Token/Character-Based Methods : - Recursive chunking with custom separator hierarchies - Precise length-based splitting Training Data Generation Automatically create domain-specific training datasets: - Generate synthetic question-answer pairs from your content - Intelligent deduplication using semantic similarity - Parallel processing for large-scale document sets - Support for local and cloud LLMs via LiteLLM Embedding Optimization Fine-tune embedding models for your specific domain: - Custom training using Sentence Transformers - Cross-platform training support (CUDA, MPS, CPU) - Dimension reduction techniques with Matryoshka Representation Learning - Comprehensive evaluation across multiple metrics - Detailed performance comparisons with baseline models Installation Usage 1. Prepare your text documents in a directory 2. Configure the pipeline in config.yaml 3. Run: To use a different configuration file, pa","default_branch":null,"files":null,"tree":[],"storefront":"/r/ALucek","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ALucek/QuicKB/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}