{"repo":"openziti/llm-gateway","free":true,"listed":false,"github":"https://github.com/openziti/llm-gateway","clone":"git clone https://github.com/openziti/llm-gateway.git","description":"Zero trust LLM gateway. OpenAI-compatible proxy with semantic routing and load balancing across OpenAI, Anthropic, Ollama, vLLM, and any compatible backend. Identity-based access, virtual API keys, and end-to-end encryption via OpenZiti","language":"Go","stars":87,"topics":["agentic-ai","ai-gateway","ai-security","anthropic","llm-gateway","llm-proxy","llm-security","load-balancing","ollama","openapi-compatible"],"license":"Apache-2.0","category":"ai-agents","readme_excerpt":"llm-gateway An OpenAI-compatible API proxy that routes requests to OpenAI, Anthropic, and any OpenAI-compatible backend (Ollama, vLLM, llama-server, SGLang, etc.). Optionally expose the gateway via zrok for zero-trust access. Why another LLM gateway? Most LLM proxies solve API translation. This one also solves the network problem: how do you connect a gateway to GPU boxes behind NAT, expose it to clients without opening ports, and route requests to the right model — all without bolting on a VPN, a service mesh, or a routing database? - Zero-trust networking with zrok (over OpenZiti) — the gateway and its backends communicate using zrok over OpenZiti overlay networks. Expose the gateway or reach a backend across NAT, air-gapped networks, or cloud boundaries without firewall rules or port forwarding. Both directions work the same way. - Semantic routing — a three-layer cascade (keyword heuristics, embedding similarity, LLM classifier) selects the best model automatically when clients omit the model field. No hand-maintained routing tables. - Multi-endpoint load balancing — weighted round-robin, health checks with passive failover, and VM sleep detection across a pool of inference servers. Works with Ollama, llama-server, vLLM, SGLang, or anything that exposes /v1/chat/completions . Built for distributing inference across real hardware. - Single binary, zero infrastructure — one Go binary, one YAML file. No database, no message queue, no sidecar. Features - OpenAI-compatible API","default_branch":null,"files":null,"tree":[],"storefront":"/r/openziti","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/openziti/llm-gateway/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}