{"repo":"erans/selfhostllm","free":true,"listed":false,"github":"https://github.com/erans/selfhostllm","clone":"git clone https://github.com/erans/selfhostllm.git","description":"A web-based calculator for estimating GPU memory requirements and maximum concurrent requests for self-hosted LLM inference.","language":"HTML","stars":50,"topics":["calculator","gpu","llm","selfhost"],"license":"MIT","category":"ai-agents","readme_excerpt":"SelfHostLLM - GPU Memory Calculator A web-based calculator for estimating GPU memory requirements and maximum concurrent requests for self-hosted LLM inference. 🔗 Live Demo : https://selfhostllm.org Overview This tool helps you calculate how many concurrent requests your GPU setup can handle when running large language models (LLMs) locally. It takes into account: - GPU VRAM capacity - Model size and parameters - Context window length - Quantization methods - KV cache overhead How It Works The Formula Calculation Breakdown 1. Total VRAM Available 2. Model Memory (Adjusted for Quantization) The model weights are loaded once and stay in memory. 3. KV Cache per Request This memory is needed for each active request's attention cache. 4. Available Memory for Inference This is what's left for KV caches after loading the model. 5. Maximum Concurrent Requests Mixture-of-Experts (MoE) Models The calculator automatically detects and handles MoE models differently: - Total Parameters : The full model size (e.g., Mixtral 8x7B = 56B total parameters) - Active Parameters : Only a subset of experts are used per token (e.g., 14B active) - Memory Calculation : Uses active memory instead of total memory for accurate estimates - Why this matters : You only need RAM for active experts, not the entire model Example : Mixtral 8x7B shows \" 94GB total, 16GB active\" - calculations use 16GB Key Assumptions - Worst-case scenario : All requests use the full context window - In reality, most requests us","default_branch":null,"files":null,"tree":[],"storefront":"/r/erans","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/erans/selfhostllm/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}