{"repo":"zli12321/LHTB","free":true,"listed":false,"github":"https://github.com/zli12321/LHTB","clone":"git clone https://github.com/zli12321/LHTB.git","description":"Long Horizon Terminal Benchmark with Dense Reward Grading","language":"Python","stars":617,"topics":["long-horizon-agents","long-horizon-tasks","claude","fable5","terminal-bench","long-horizon-terminal-bench","gpt-56-sol","deepseek-v4-pro","grok-45","kimi-k3"],"license":"Apache-2.0","category":"security-tools","readme_excerpt":"Long-Horizon Terminal-Bench (LHTB) English 简体中文 Long-Horizon Terminal-Bench is a 46-task benchmark for measuring how well LLM agents sustain useful work in a containerized terminal over hundreds of steps . Unlike short-horizon coding benchmarks where an agent writes one artifact and stops, LHTB drops the agent into a stateful environment and grades it with hidden, rebuild-from-artifact verifiers — self-reported progress does not count. Tasks span interactive games & puzzles, multimodal analysis, software / reverse engineering, scientific computing, earth & energy systems, security & performance, research reproduction, and professional APEX-style workflows. Companion to Terminal-Bench / Terminal-Bench 2.0. Evaluated with Harbor. ⚠️ Harness update — continue-until-timeout. LHTB adds one behavior on top of Harbor: for long-horizon tasks the agent keeps working until the task timeout instead of ending the moment it declares the task complete. After the agent stops, the harness runs the hidden verifier and — if it hasn't fully passed — resumes the agent with a binary rejection only , repeating until the timeout elapses or the verifier passes. Failure strings, scalar rewards, test output, paths, and gate counts are not disclosed. This is controlled by continue until timeout = true in the [agent] block of a task's task.toml (set on 30 of the 46 tasks). Maintainers may opt into diagnostic feedback with HB VERIFIER FEEDBACK MODE=diagnostic , but such runs are not benchmark-comparable;","default_branch":null,"files":null,"tree":[],"storefront":"/r/zli12321","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/zli12321/LHTB/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}