{"repo":"vincentkoc/airgapped-offfline-rag","free":true,"listed":false,"github":"https://github.com/vincentkoc/airgapped-offfline-rag","clone":"git clone https://github.com/vincentkoc/airgapped-offfline-rag.git","description":"Secure, locally-run Retrieval-Augmented Generation system for document-based question-answering, utilizing Llama 3, Mistral, and Gemini models with a user-friendly Streamlit interface.","language":"Python","stars":87,"topics":["chromadb","docker","langchain","llama","llm","machine-learning","mistral","offline","privacy","rag"],"license":"GPL-3.0","category":"ai-agents","readme_excerpt":"Airgapped Offline RAG This project by Vincent Koc implements a Retrieval-Augmented Generation (RAG) based Question-Answering system for documents. It uses Llama 3, Mistral, and Gemini models for local inference with LlaMa c++, langchain for orchestration, chromadb for vector storage, and Streamlit for the user interface. Table of Contents - Airgapped Offline RAG - Table of Contents - Setup - Running the Application - Locally - Using Docker - Usage - Configuration - Features - Supported Features - Future Features - Contributing - License - Acknowledgments Setup 1. Ensure Python 3.9 is installed : You can use pyenv : 2. Create a virtual environment and install dependencies : 3. Download Models : Download the Llama 3 (8B) and Mistral (7B) models in GGUF format and place them in the models/ directory. TheBloke on Hugging Face has shared the models here: - Mistral-7B-Instruct-v0.2-GGUF - LLaMA-Pro-8B-Instruct-GGUF The models from unsloth have also been tested and can be found here: - Gemma-2-2b-it.q2 k.gguf - Llama-3.2-3B-Instruct-Q2 K.gguf 4. Qdrant Sentence Transformer Model : This will be downloaded automatically on the first run. If running the airgapped RAG locally, it's best to run the codebase with internet access initially to download the model. Running the Application Locally Using Docker Usage 1. Upload PDF documents using the file uploader. 2. Select the model you want to use (e.g., Mistral). 3. Enter your question in the text input. 4. Click \"Generate Answer\" to get a ","default_branch":null,"files":null,"tree":[],"storefront":"/r/vincentkoc","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/vincentkoc/airgapped-offfline-rag/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}