{"repo":"EricLBuehler/mistral.rs","free":true,"listed":false,"github":"https://github.com/EricLBuehler/mistral.rs","clone":"git clone https://github.com/EricLBuehler/mistral.rs.git","description":"Fast, flexible LLM inference","language":"Rust","stars":7611,"topics":["llm","rust","uqff"],"license":"MIT","category":"ai-agents","readme_excerpt":"Documentation Quickstart Supported models Rust SDK Python SDK Discord Latest - Muse Glimmer 30B : native text, image, and video inference with ATEM tool calling, reasoning controls, LoRA, ISQ/UQFF, and companion-projector GGUF loading. Model notes - GGUF loading : load a local file with -f , or select a published artifact with --quant . Tokenizer, configuration, and multimodal projector files are discovered when the available metadata identifies them unambiguously. Guide - OpenAI-compatible Skills : upload /v1/skills bundles and reference them from Responses requests for reusable procedures, helper scripts, and local data. Guide - OpenAI-compatible file inputs : upload /v1/files , attach Responses input file or Chat file parts, and mount request files into shell/code sessions. Guide - DiffusionGemma : block-diffusion text generation. Fully integrated: paged attention, prefix caching, ISQ, multimodal, and tool calling. Guide - Anthropic Messages API : mistralrs serve now exposes Anthropic-compatible /v1/messages and /v1/messages/count tokens endpoints alongside the OpenAI-compatible /v1 API. Guide - v0.8.2 CUDA performance : paged-attention and MoE optimizations deliver strong results on GB10, B200, and H100 SXM. Benchmarks - Agentic runtime : web search, local Python code execution, shell execution, OpenAI-compatible Skills, session management, and custom tool hooks. Guide - Gemma 4 : full multimodal: text, image, video, and audio input. Supported models Video setup Benchmark","default_branch":null,"files":null,"tree":[],"storefront":"/r/EricLBuehler","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/EricLBuehler/mistral.rs/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}