{"repo":"sensoris/semcache","free":true,"listed":false,"github":"https://github.com/sensoris/semcache","clone":"git clone https://github.com/sensoris/semcache.git","description":"Semantic caching layer for your LLM applications. Reuse responses and reduce token usage.","language":"Rust","stars":96,"topics":["anthropic","gemini","genai","llm","openai","semantic-cache"],"license":"MIT","category":"ai-agents","readme_excerpt":"⚡ semcache semcache is a semantic caching layer for your LLM applications. - semcache website - semcache docs Quick Start Start the Semcache Docker image: Configure your application e.g with the OpenAI Python SDK: Node.js follows a similar pattern of changing the base URL to point to your Semcache host: Features - 🧠 Completely in-memory - Prompts, responses and the vector database are stored in-memory - 🎯 Flexible by design - Can work with your custom or private LLM APIs - 🔌 Support for major LLM APIs - OpenAI, Anthropic, Gemini, and more - ⚡ HTTP proxy mode - Drop-in replacement that reduces costs and latency - 📈 Prometheus metrics - Full observability out of the box - 📊 Build-in dashboard - Monitor cache performance at /admin - 📤 Smart eviction - LRU cache eviction policy Semcache is still in beta and being actively developed. How it works Semcache accelerates LLM applications by caching responses based on semantic similarity. When you make a request Semcache first searches for previously cached answers to similar prompts and delivers them immediately. This eliminates redundant API calls, reducing both latency and costs. Semcache also operates in a \"cache-aside\" mode, allowing you to load prompts and responses yourself. Example Integrations For comprehensive provider configuration and detailed code examples, visit our LLM Providers & Tools documentation. HTTP Proxy Point your existing SDK to Semcache instead of the provider's endpoint. OpenAI Anthropic LangChain LiteL","default_branch":null,"files":null,"tree":[],"storefront":"/r/sensoris","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/sensoris/semcache/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}