{"repo":"mburaksayici/smallevals","free":true,"listed":false,"github":"https://github.com/mburaksayici/smallevals","clone":"git clone https://github.com/mburaksayici/smallevals.git","description":"smallevals — CPU-fast, GPU-blazing fast offline retrieval evaluation for RAG systems with tiny QA models.","language":"Python","stars":23,"topics":["offline-evaluation","qa","qa-generation","question-generation","rag-evaluation","retrieval-evaluation","retrieval-metrics","tiny-llm","vector-database","chroma"],"license":null,"category":"databases-storage","readme_excerpt":"smallevals - Local LLM Evaluation Framework with Tiny 0.6B Models A lightweight evaluation framework powered by tiny ( really tiny ) 0.6B models — runs 100% locally on CPU/GPU/MPS, attach any vector DB connection and run, fast and free. Evaluation tools requiring LLM-as-a-judge or external, that costs/doesn't scale easily. evaluates in seconds in GPU, in minutes in any CPU ! Evaluate Retrieval Evaluation of RAG system includes retrieval and RAG stage, attacks to test retrieval and RAG answers(in the near future)! Models Model Name Task Status Link ------------ ------- -------- ------ QAG-0.6B Generate golden Q/A from chunks or docs (synthetic evaluation data) Available 🤗 CRC-0.6B Context relevance classifier (question ↔ retrieved chunk) Incoming — GJ-0.6B Groundedness / faithfulness judge (answer ↔ context) Incoming — ASM-0.6B Answer correctness / semantic similarity Incoming — Current Focus : Retrieval evaluation (QAG-0.6B), after being sure the model generates correct answers and better questions for RAG(it does, but still room for improvement), the model will be the first model of pipeline leading to (RAG) generation evaluation models (CRC-0.6B, GJ-0.6B, ASM-0.5B) which are the future work. Installation Quick Start Evaluate Retrieval Quality (Python) Connect to your favourite Vector DB (Milvus, Elastic, PGVector, Chroma, Pinecone, FAISS, Weawiate), attach your favourite embeddings, generate questions, and visualise results! Under the hood, generates question per chunk, an","default_branch":null,"files":null,"tree":[],"storefront":"/r/mburaksayici","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mburaksayici/smallevals/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}