{"repo":"Mattbusel/tokio-prompt-orchestrator","free":true,"listed":false,"github":"https://github.com/Mattbusel/tokio-prompt-orchestrator","clone":"git clone https://github.com/Mattbusel/tokio-prompt-orchestrator.git","description":"Multi-core, Tokio-native orchestration for LLM pipelines.","language":"Rust","stars":57,"topics":["llm","orchestration","rust","tokio","async","circuit-breaker","agent","llm-inference","production","backpressure"],"license":"MIT","category":"workflow-automation","readme_excerpt":"tokio-prompt-orchestrator Production-grade, multi-core Tokio orchestration for LLM inference pipelines. Five-stage bounded-backpressure DAG with deduplication, circuit breakers, rate limiting, prompt injection/jailbreak detection , provider arbitrage (cheapest provider meeting your latency SLA), adaptive worker pool sizing , and an optional autonomous self-improving control loop. Supports Anthropic, OpenAI, llama.cpp, vLLM, and any custom backend. Exposes REST, WebSocket, SSE, MCP (Claude Desktop), Prometheus metrics, and OpenTelemetry distributed tracing. --- What's New v1.9.0 — Plugin System and Request Deduplication Plugin System The new Round-7 plugin API in src/plugin.rs adds a full Plugin trait–based extension system that runs before and after inference. Key types: Plugin (trait), PluginV2Chain , PluginV2Registry , PluginError , PluginInfo Built-in plugins: - ProfanityFilterPlugin — blocks requests containing a configurable word list (case-insensitive) - ResponseLengthCapPlugin { max tokens } — truncates response token lists exceeding the cap, returns PluginError::ResponseModified - LatencyLoggerPlugin — records per-request latency samples to a shared Vec via record latency(ms) Features: - Plugin::on request(&mut PromptRequest) — intercept and optionally reject requests before inference - Plugin::on response(&mut Vec ) — inspect/mutate response tokens after inference - PluginV2Registry::register(Box ) — ordered plugin execution in registration order - PluginV2Registry::","default_branch":null,"files":null,"tree":[],"storefront":"/r/Mattbusel","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Mattbusel/tokio-prompt-orchestrator/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}