{"repo":"mclenhard/mcp-evals","free":true,"listed":false,"github":"https://github.com/mclenhard/mcp-evals","clone":"git clone https://github.com/mclenhard/mcp-evals.git","description":"A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.","language":"TypeScript","stars":132,"topics":["ai","evals","mcp"],"license":"MIT","category":"mcp-servers","readme_excerpt":"MCP Evals A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring, with built-in observability support . This helps ensure your MCP server's tools are working correctly, performing well, and are fully observable with integrated monitoring and metrics. Installation As a Node.js Package As a GitHub Action Add the following to your workflow file: Usage -- Evals 1. Create Your Evaluation File You can create evaluation configurations in either TypeScript or YAML format. Option A: TypeScript Configuration Create a file (e.g., evals.ts ) that exports your evaluation configuration: Option B: YAML Configuration For simpler configuration, you can use YAML format (e.g., evals.yaml ): 2. Run the Evaluations As a Node.js Package You can run the evaluations using the CLI with either TypeScript or YAML files: As a GitHub Action The action will automatically: 1. Run your evaluations 2. Post the results as a comment on the PR 3. Update the comment if the PR is updated Evaluation Results Each evaluation returns an object with the following structure: Configuration Environment Variables - OPENAI API KEY : Your OpenAI API key (required for OpenAI models) - ANTHROPIC API KEY : Your Anthropic API key (required for Anthropic models) [!NOTE] If you're using this GitHub Action with open source software, enable data sharing in the OpenAI billing dashboard to claim 2.5 million free GPT-4o mini tokens per day, making this Action effect","default_branch":null,"files":null,"tree":[],"storefront":"/r/mclenhard","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mclenhard/mcp-evals/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}