{"repo":"khromov/svelte-bench","free":true,"listed":false,"github":"https://github.com/khromov/svelte-bench","clone":"git clone https://github.com/khromov/svelte-bench.git","description":"An LLM benchmark for Svelte 5 based on the OpenAI methodology from OpenAIs paper \"Evaluating Large Language Models Trained on Code\".","language":"TypeScript","stars":186,"topics":["svelte","sveltekit","ai"],"license":null,"category":"ai-agents","readme_excerpt":"SvelteBench An LLM benchmark for Svelte 5 based on the HumanEval methodology from OpenAI's paper \"Evaluating Large Language Models Trained on Code\". This benchmark evaluates LLMs' ability to generate functional Svelte 5 components with proper use of runes and modern Svelte features. Overview SvelteBench evaluates LLM-generated Svelte components by testing them against predefined test suites. It works by sending prompts to LLMs, generating Svelte components, and verifying their functionality through automated tests. The benchmark calculates pass@k metrics (typically pass@1 and pass@10) to measure model performance. Supported Providers SvelteBench supports multiple LLM providers: - OpenAI - Anthropic - Google Gemini - OpenRouter - Access to models through a unified API - Fireworks - Fireworks-hosted models through its OpenAI-compatible API - Ollama - Run models locally - Z.ai - Moonshot - xAI - Meta - Cursor - MiniMax - MiniMax's OpenAI-compatible API Setup The recommended workflow is to choose a provider and configure its API key from the TUI. Start it with: If the selected provider has no saved key, the TUI prompts for one and stores it in .env for future runs. You only need to configure the providers you use. For scripted or CI usage, create .env from the example and add your keys: The existing environment variables remain supported by pnpm run-tests and the standard pnpm start command. Running the Benchmark TUI Execution The TUI guides you through: 1. Selecting a provider 2","default_branch":null,"files":null,"tree":[],"storefront":"/r/khromov","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/khromov/svelte-bench/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}