{"repo":"Picovoice/picollm","free":true,"listed":false,"github":"https://github.com/Picovoice/picollm","clone":"git clone https://github.com/Picovoice/picollm.git","description":"On-device LLM Inference Powered by X-Bit Quantization","language":"Python","stars":317,"topics":["llm","compression","efficient-inference","gemma","generative-ai","language-model","language-models","large-language-model","llama","llama2"],"license":"Apache-2.0","category":"ai-agents","readme_excerpt":"picoLLM Inference Engine Made in Vancouver, Canada by Picovoice picoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running compressed large language models. picoLLM Inference Engine is: - Accurate; picoLLM Compression improves GPTQ by significant margins - Private; LLM inference runs 100% locally. - Cross-Platform - Linux (x86 64), macOS (arm64, x86 64), and Windows (x86 64, arm64) - Raspberry Pi (3, 4, 5) - Android and iOS - Chrome, Safari, Edge, and Firefox - Runs on CPU and GPU - Free for open-weight models Table of Contents - picoLLM - Table of Contents - Showcases - Raspberry Pi - Android - iOS - Cross-Browser Local LLM - Llama-3-70B-Instruct on GeForce RTX 4090 - Local LLM-Powered Voice Assistant on Raspberry Pi - Local Llama-3-8B-Instruct Voice Assistant on CPU - Accuracy - Models - AccessKey - Demos - Python - .NET - Node.js - Android - iOS - Web - C - SDKs - Python - .NET - Node.js - Android - iOS - Web - C - Releases Showcases Raspberry Pi Android iOS Cross-Browser Local LLM Live Demo — Works offline! Llama-3-70B-Instruct on GeForce RTX 4090 Local LLM-Powered Voice Assistant on Raspberry Pi Local Llama-3-8B-Instruct Voice Assistant on CPU Accuracy picoLLM Compression is a novel large language model (LLM) quantization algorithm developed within Picovoice. Given a task-specific cost function, picoLLM Compression automatically learns the optimal bit allocation strategy across and within LLM's weights. Existing techniques require a fixed ","default_branch":null,"files":null,"tree":[],"storefront":"/r/Picovoice","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Picovoice/picollm/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}