{"repo":"colomalabs/coloma","free":true,"listed":false,"github":"https://github.com/colomalabs/coloma","clone":"git clone https://github.com/colomalabs/coloma.git","description":"The Continuous Verification and Optimization layer for self-hosted LLMs","language":"Python","stars":13,"topics":["llm-inference","mlops","self-hosted","vllm","vllm-serving"],"license":"Apache-2.0","category":"self-hosted-apps","readme_excerpt":"Coloma The Continuous Verification and Optimization layer for self-hosted LLMs Features - Tune vLLM deployments to maximize the metrics you care about, and prevent runtime crashes. - Compare your tuned models, and deploy them. - Drop-in OpenAI-compatible proxy: sits between your workload and your endpoint. - Live and saved traffic inspection. - Image optimization: optimizes base64 images before they hit your GPU, shrinking payload with no client changes. - Structured-output schema validation and custom Python validators. - Idle-compute verification: re-runs a heavier check on past predictions when the proxy is idle, no added latency. - Chat interface. - Runs 100% on-prem, no data collected / sent to third-party , except if you use an LLM provider instead of a self-hosted model. Prerequisites - uv - node 24 - Docker Engine - NVIDIA Container Toolkit - (Optional) Poppler for PDF import Quickstart Then head to Profile & deploy, after profiling and deploying, enjoy the Chat tab, proxy for your OpenAI client, and Traffic tab. Architecture Configuration - COLOMA API KEY protects both the dashboard and proxy - If you're using Coloma to deploy models, the API key field in Proxy settings is used to protect the OpenAI endpoint of your deployed model - If you're using Coloma to point to LLM providers (e.g. OpenAI API), set your OpenAI API key in Proxy settings Roadmap - Better support for VLM profiling. - Reasoning models support, with higher reasoning effort on verification passes. - L","default_branch":null,"files":null,"tree":[],"storefront":"/r/colomalabs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/colomalabs/coloma/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}