{"repo":"bigcode-project/bigcodearena","free":true,"listed":false,"github":"https://github.com/bigcode-project/bigcodearena","clone":"git clone https://github.com/bigcode-project/bigcodearena.git","description":"BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution","language":"Python","stars":61,"topics":["code-generation","llm","vibe-coding"],"license":"Apache-2.0","category":"dev-tools","readme_excerpt":"BigCodeArena 🌸 About • 📦 Installation • 🏆 BigCodeReward • 🎯 AutoCodeArena • 🌐 HF Space • 🔧 Configuration • 📜 Citation 🌸 About BigCodeArena is an open human evaluation platform for code generation, built on top of Chatbot Arena with a comprehensive and on-the-fly execution environment. Unlike traditional evaluation platforms, BigCodeArena enables the execution of LLM-generated code and allows humans to interact with execution processes and outcomes, addressing the challenge of manually examining code quality. 📦 Installation Prerequisites - Python 3.8+ - Docker (required for AutoCodeArena code execution) - API keys for LLM providers (OpenAI, Anthropic, etc.) Docker Setup (for AutoCodeArena) 🏆 BigCodeReward Purpose : Evaluate reward model consistency with human preferences on code generation tasks BigCodeReward uses 4.7K+ postprocessed conversations from BigCodeArena to systematically evaluate how well LLM-based reward models align with actual human preferences in code generation scenarios. Evaluation Pipeline : 1. Evaluate - Judge code comparisons using LLM-as-a-Judge (with/without execution) 2. Analyze - Compute performance metrics and consistency scores 3. Correlate - Calculate ELO ratings and correlation with human votes Evaluation Modes : - With Execution : Judge models see code + execution results + screenshots - Without Execution : Judge models see code only (code-centric evaluation) Quick Start Evaluate reward model alignment with human preferences: Step 1: Set","default_branch":null,"files":null,"tree":[],"storefront":"/r/bigcode-project","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/bigcode-project/bigcodearena/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}