{"repo":"aisa-group/PostTrainBench","free":true,"listed":false,"github":"https://github.com/aisa-group/PostTrainBench","clone":"git clone https://github.com/aisa-group/PostTrainBench.git","description":"Measuring how well CLI agents like Claude Code or Codex CLI can post-train base LLMs on a single H100 GPU in 10 hours","language":"Python","stars":518,"topics":["ai-safety","claude-code","codex-cli","gemini-cli","post-training","ai-research-automation"],"license":"MIT","category":"ai-agents","readme_excerpt":"PostTrainBench: Can LLM Agents Automate LLM Post-Training? We introduce PostTrainBench, a benchmark that measures the ability of CLI agents to post-train pre-trained large language models (LLMs). In PostTrainBench, the agent's task is to improve the performance of a base LLM on a given benchmark. The agent is given access to an evaluation script and 10 hours on an H100 GPU. Performance is measured by the benchmark score of the post-trained LLM. This setup naturally evaluates an agent's ability to conduct AI R&D. [!IMPORTANT] Harbor support coming soon! This repository currently targets our internal HPC cluster (HTCondor). We are adding Harbor support to make it straightforward to run on rented hardware (e.g., cloud GPUs). See our PR. Leaderboard Scores are weighted averages across 7 benchmarks and 4 models (Qwen3-1.7B, Qwen3-4B, SmolLM3-3B, and Gemma-3-4B). Agents with multiple runs show averaged results. Rank Agent Scaffold Avg AIME 2025 Arena Hard BFCL GPQA GSM8K HealthBench HumanEval ---: --- --- ---: ---: ---: ---: ---: ---: ---: ---: - Official Instruct Models - 51.1 29.2 70.2 85.0 36.2 87.0 43.3 71.5 1 Opus 4.6 Claude Code 23.2 5.0 7.8 75.9 25.5 41.0 18.8 24.7 2 Gemini 3.1 Pro OpenCode 21.6 3.9 7.4 62.8 18.5 45.5 14.5 40.2 3 GPT-5.2 Codex CLI 21.4 0.8 6.6 52.5 23.7 55.9 15.8 30.2 4 GPT 5.4 Codex CLI 20.2 0.6 10.1 31.1 28.0 48.2 17.3 27.3 5 GPT 5.1 Codex Max Codex CLI 19.7 0.6 4.0 30.8 24.0 51.6 17.8 32.0 6 Gemini 3 Pro Gemini CLI 18.1 1.7 6.3 42.3 21.2 39.1 17.3 22.7 7 ","default_branch":null,"files":null,"tree":[],"storefront":"/r/aisa-group","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/aisa-group/PostTrainBench/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}