{"repo":"InftyAI/llmaz","free":true,"listed":false,"github":"https://github.com/InftyAI/llmaz","clone":"git clone https://github.com/InftyAI/llmaz.git","description":"☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!","language":"Go","stars":308,"topics":["kubernetes","llm","llamacpp","sglang","vllm","text-generation-inference","huggingface","modelscope","ollama","inference"],"license":"Apache-2.0","category":"deployment-docker-iac","readme_excerpt":"Easy, advanced inference platform for large language models on Kubernetes [![GoReport Widget]][GoReport Status] [GoReport Widget]: https://goreportcard.com/badge/github.com/inftyai/llmaz [GoReport Status]: https://goreportcard.com/report/github.com/inftyai/llmaz llmaz (pronounced /lima:z/ ), aims to provide a Production-Ready inference platform for large language models on Kubernetes. It closely integrates with the state-of-the-art inference backends to bring the leading-edge researches to cloud. 🌱 llmaz is alpha now, so API may change before graduating to Beta. Overview Architecture Key Features - Easy of Use : People can quick deploy a LLM service with minimal configurations. - Broad Backends Support : llmaz supports a wide range of advanced inference backends for different scenarios, like vLLM, Text-Generation-Inference, SGLang, llama.cpp, TensorRT-LLM. Find the full list of supported backends here. - Heterogeneous Cluster Support : llmaz supports serving the same LLM with heterogeneous devices together with InftyAI Scheduler for the sake of cost and performance. - Various Model Providers : llmaz supports a wide range of model providers, such as HuggingFace, ModelScope, ObjectStores. llmaz will automatically handle the model loading, requiring no effort from users. - Distributed Inference : Multi-host & homogeneous xPyD support with LWS from day 0. Will implement the heterogeneous xPyD in the future. - AI Gateway Support : Offering capabilities like token-based rate limit","default_branch":null,"files":null,"tree":[],"storefront":"/r/InftyAI","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/InftyAI/llmaz/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}