{"repo":"anaslimem/llm-gateway-core","free":true,"listed":false,"github":"https://github.com/anaslimem/llm-gateway-core","clone":"git clone https://github.com/anaslimem/llm-gateway-core.git","description":"A production-grade LLM gateway that abstracts multiple model providers, implements intelligent routing, caching, retries, and observability to deliver reliable, cost-aware LLM access.","language":"Python","stars":18,"topics":["fastapi","ai-infrastructure","google-gemini","grafana","ollama","prometheus","rate-limiting","redis-cache"],"license":"MIT","category":"analytics","readme_excerpt":"LLM Gateway Core LLM Gateway Core is a production-grade infrastructure component designed to abstract multiple Large Language Model (LLM) providers behind a single, unified API. It implements intelligent routing, distributed caching, atomic rate limiting, and comprehensive observability to provide reliable and cost-effective LLM access. System Architecture The gateway is built on a high-performance FastAPI backend, utilizing a provider-agnostic interface that allows for seamless integration of both cloud-based and local model providers. Core Components API Layer : FastAPI-based REST API providing standardized chat completion endpoints. Provider Router : Dynamically selects the optimal model provider based on request hints (online, local, fast, secure). Redis Integration : Distributed Cache : Persistently stores provider responses to reduce latency and API costs. Rate Limiter : Implements a token bucket algorithm via Redis Lua scripts for atomic, distributed request throttling. Monitoring Stack : Full observability with Prometheus for metrics collection and Grafana for visualization. Streamlit Frontend : A clean, responsive interface for demonstration and testing purposes. Integrated Providers The gateway currently supports the following providers: Google Gemini : High-performance cloud integration for 'online' and 'fast' request modes. Ollama : Local integration for 'local' and 'secure' request modes, enabling private, on-premise inference. User Interface The Streamlit fronte","default_branch":null,"files":null,"tree":[],"storefront":"/r/anaslimem","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/anaslimem/llm-gateway-core/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}