{"repo":"200lz/llm-engineering-platform","free":true,"listed":false,"github":"https://github.com/200lz/llm-engineering-platform","clone":"git clone https://github.com/200lz/llm-engineering-platform.git","description":"A production-oriented LLM engineering platform with OpenAI-compatible serving, streaming, observability, evaluation, reproducible experiments, and deterministic RAG.","language":"Python","stars":17,"topics":["ai-infrastructure","docker","evaluation","experiment-tracking","fastapi","llama-cpp","llm","llmops","prometheus","python"],"license":"MIT","category":"ai-agents","readme_excerpt":"LLM Production Platform A production-oriented, CPU-first LLM engineering platform built around FastAPI and llama.cpp. It demonstrates the systems surrounding inference—streaming, compatibility, observability, evaluation, reproducibility, deployment, and retrieval—without reimplementing the inference engine. The repository is deliberately local, inspectable, and educational. Default tests require no model, GPU, Docker daemon, hosted API, database, or secret. Project overview The platform has four independently usable surfaces: - an OpenAI-shaped FastAPI gateway for a separately running llama-server ; - llm-eval , a deterministic evaluation and regression CLI; - llm-experiment , an immutable local experiment registry; and - llm-rag , a deterministic local retrieval engineering toolkit. The serving runtime does not import the evaluation, experiment, or RAG packages. Each surface has explicit persistence, failure, privacy, and reproducibility contracts. Why this project exists Running a model is only one part of production LLM engineering. The surrounding system must define what happens when streams fail, clients disconnect, upstream data is malformed, experiments drift, artifacts change, or retrieval results cannot be traced to source text. This project makes those concerns visible in a compact codebase: - protocol and backend wire formats stay separated; - streaming preserves backpressure and cancellation; - metrics use bounded-cardinality labels; - evaluations and regressions ","default_branch":null,"files":null,"tree":[],"storefront":"/r/200lz","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/200lz/llm-engineering-platform/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}