{"repo":"jimmc414/claudecode_gemini_and_codex_swebench","free":true,"listed":false,"github":"https://github.com/jimmc414/claudecode_gemini_and_codex_swebench","clone":"git clone https://github.com/jimmc414/claudecode_gemini_and_codex_swebench.git","description":"Toolkit for measuring Claude Code and Codex performance over time against a baseline using SWEbench-lite dataset **No API key required for Max or Pro subscribers**","language":"Python","stars":32,"topics":["claude-code","claudecode","eval","evaluation-framework","swebench"],"license":null,"category":"analytics","readme_excerpt":"SWE-bench Code Model Performance Monitor Purpose This project provides an empirical framework for measuring the performance of code-focused language models like Claude Code, Codex, and Gemini on real-world software engineering tasks. It was built to provide objective, reproducible metrics that allow users to assess these tools for themselves, rather than relying on anecdotal reports or marketing claims. The SWE-bench benchmark presents the model with actual GitHub issues from popular open-source projects and measures its ability to generate patches that successfully resolve these issues. This provides a concrete, measurable answer to the question: \"How well do these code models actually perform on real software engineering tasks?\" Platform support: The tools in this repository run on Linux, macOS, and Windows (including WSL). Replace python with python3 on Unix-like systems or py on Windows if needed. Getting Started in 5 Minutes For detailed setup instructions, see Prerequisites and Installation below. Quick Start (After Installation) Prerequisites Before starting, ensure you have: 1. Python 3.8 or newer 2. Claude Code, Codex, or Gemini CLI installed and logged in 3. Docker installed and running - Needs 50GB free disk space for images - 16GB+ RAM recommended - For Mac/Windows: Increase Docker Desktop memory to 8GB+ Don't have Docker? The easiest way is to ask Claude Code to set it up: Or see Manual Docker Setup below. Installation Troubleshooting Setup If you get errors: - \"","default_branch":null,"files":null,"tree":[],"storefront":"/r/jimmc414","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jimmc414/claudecode_gemini_and_codex_swebench/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}