{"repo":"Lightning-AI/LitServe","free":true,"listed":false,"github":"https://github.com/Lightning-AI/LitServe","clone":"git clone https://github.com/Lightning-AI/LitServe.git","description":"A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.","language":"Python","stars":3929,"topics":["ai","api","serving","artificial-intelligence","deep-learning","developer-tools","fastapi","rest-api","web"],"license":"Apache-2.0","category":"api-integrations-sdks","readme_excerpt":"Build custom inference servers in pure Python Define exactly how inference works for models, agents, RAG, or pipelines. Control batching, routing, streaming, and orchestration without MLOps glue or config files.   ✅ Custom inference logic ✅ 2× faster than FastAPI ✅ Agents, RAG, pipelines, more ✅ Custom logic + control ✅ Any PyTorch model ✅ Self-host or managed ✅ Multi-GPU autoscaling ✅ Batching + streaming ✅ BYO model or vLLM ✅ No MLOps glue code ✅ Easy setup in Python ✅ Serverless support Quick start • Examples • Features • Performance • Hosting • Docs     Why LitServe? Most serving tools (vLLM, etc..) are built for a single model type and enforce rigid abstractions. They work well until you need custom logic, multiple models, agents, or non standard pipelines. LitServe lets you write your own inference engine in Python. You define how requests are handled, how models are loaded, how batching and routing work, and how outputs are produced. LitServe handles performance, concurrency, scaling, and deployment. Use LitServe to build inference APIs, agents, chatbots, RAG systems, MCP servers, or multi model pipelines. Run it locally, self host anywhere, or deploy with one click on Lightning AI.   Want the easiest way to host inference? Over 380,000 developers use Lightning Cloud, the simplest way to run LitServe without managing infrastructure. Deploy with one command, get autoscaling GPUs, monitoring, and a free tier. No cloud setup required. Or self host anyw","default_branch":null,"files":null,"tree":[],"storefront":"/r/Lightning-AI","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Lightning-AI/LitServe/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}