{"repo":"arthi-arumugam-git/whatbroke","free":true,"listed":false,"github":"https://github.com/arthi-arumugam-git/whatbroke","clone":"git clone https://github.com/arthi-arumugam-git/whatbroke.git","description":"Diff your AI agent's behavior between two runs. See exactly which tool calls, args, costs and outputs changed when you swap models or edit prompts.","language":"TypeScript","stars":19,"topics":["ai-agents","anthropic","diff","evals","llm","openai","regression-testing","tracing","agent-testing","developer-tools"],"license":"MIT","category":"ai-agents","readme_excerpt":"whatbroke Diff your AI agent's behavior between two runs. English · 简体中文 · 日本語 · 한국어 · Español · Português · Français · Deutsch · Русский · हिन्दी Swap a model, tweak a prompt, bump a framework version, then run whatbroke and see exactly what changed: which tool calls disappeared, which arguments drifted, where cost and latency moved, and which outputs flipped. Text diffs can't see this. Your agent can say \"your subscription is cancelled\" while silently skipping the cancel subscription call. The words look fine. The behavior broke. That's a real diff from swapping to a 3x smaller model, recorded on a laptop with ollama. The replies kept passing the vibe check while the model started sending its tool's own JSON schema as the tool arguments and skipping the actual cancellation call. For the full story and the commands to reproduce it, see the case study: what a 3x smaller model changed in a tool-calling agent. There's a second one on swapping vendors at the same size: a same-size vendor swap changed more than the 3x downgrade, and a third on upgrading a generation: the model that fixed the fake IDs also stopped issuing refunds. The fourth one changes no model at all. Same weights, same prompts, same machine, only the path the request takes: the proxy that silently turned an agent into a chatbot. One LiteLLM route with streaming on dropped every tool call in every scenario, returned HTTP 200 and finish reason: \"stop\" , and printed the tool call as prose to the customer. If you'r","default_branch":null,"files":null,"tree":[],"storefront":"/r/arthi-arumugam-git","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/arthi-arumugam-git/whatbroke/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}