{"repo":"Metabuilder-Labs/tokenjam-bench","free":true,"listed":false,"github":"https://github.com/Metabuilder-Labs/tokenjam-bench","clone":"git clone https://github.com/Metabuilder-Labs/tokenjam-bench.git","description":"Benchmarking & Evals for TokenJam","language":"Python","stars":12,"topics":["agentic-ai","ai","cli","duckdb","hermes","hermes-agent","llm","ml","openclaw","python"],"license":"MIT","category":"machine-learning","readme_excerpt":"Evidence-based Benchmarking & Evaluations for TokenJam recommendations Does TokenJam's \"downsize this model\" recommendation hold up? TokenJam Bench runs the cheaper model against executable benchmarks and tells you (with statistics) where it keeps up and where it breaks and validates if the savings are true. Don't have pipx? brew install pipx on macOS, apt install pipx on Debian/Ubuntu. pip install \"tokenjam-bench[providers,datasets]\" also works in a clean venv. No cloud · No signup · Runs Locally --- It answers the one question TokenJam itself can't: when TokenJam says \"downsize this model,\" does the cheaper model still get the work right — and how much does it actually save? --- How it's a proof of TokenJam (not a generic model comparison) - The cheaper candidate is the model TokenJam's own downsize analyzer would route to — pulled live from tokenjam.core.optimize.DOWNGRADE CANDIDATES . - Cost is priced with TokenJam's own pricing table ( tokenjam.core.pricing.get rates ) — same dollars TokenJam reports. - Accuracy is the benchmark pass-rate against real test suites — a measurement , never a judgment. - Every result is stamped with the exact TokenJam version under test. --- Real evidence: downsizing Claude to Haiku breaks code, not math The 2026-06-26 multipair run benchmarked 18 model-pair × suite configurations at real measured rates (tokenjam 0.5.2). The sharpest result comes from routing claude-opus-4-7 down to claude-haiku-4-5 . The cheaper model costs far less. It hol","default_branch":null,"files":null,"tree":[],"storefront":"/r/Metabuilder-Labs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Metabuilder-Labs/tokenjam-bench/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}