{"repo":"intentee/paddler","free":true,"listed":false,"github":"https://github.com/intentee/paddler","clone":"git clone https://github.com/intentee/paddler.git","description":"Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.","language":"Rust","stars":1657,"topics":["ai","llamacpp","llm","llmops","load-balancer"],"license":"Apache-2.0","category":"networking-infra","readme_excerpt":"Paddler Digital products and their users need privacy, reliability, cost control, and an option to be independent from closed-source model providers. Paddler is an open-source LLM load balancer and serving platform. It allows you to run inference, deploy, and scale LLMs on your own infrastructure, providing a great developer experience along the way. Key features Inference through a built-in llama.cpp engine LLM-specific load balancing Works through agents that can be added dynamically, allowing integration with autoscaling tools Request buffering, enabling scaling from zero hosts Dynamic model swapping Built-in web admin panel for management, monitoring, and testing Observability metrics Who is Paddler for? Product teams that need LLM inference and embeddings in their features DevOps/LLMOps teams that need to run and deploy LLMs at scale Organizations handling sensitive data with high compliance and privacy requirements (medical, financial, etc.) Organizations wanting to achieve predictable LLM costs instead of being exposed to per-token pricing Product leaders who need reliable model performance to maintain a consistent user experience of their AI-based features Community - Discord https://discord.gg/92x3Z8a4gj - Reddit (just started a subreddit, we will see how it goes :)) https://www.reddit.com/r/paddler/ Installation and Quickstart Paddler is self-contained in a single binary file, so all you need to do to start using it is obtain the paddler binary and make it available","default_branch":null,"files":null,"tree":[],"storefront":"/r/intentee","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/intentee/paddler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}