{"repo":"lastmile-ai/mcp-eval","free":true,"listed":false,"github":"https://github.com/lastmile-ai/mcp-eval","clone":"git clone https://github.com/lastmile-ai/mcp-eval.git","description":"Lightweight eval framework for MCP servers, built on mcp-agent","language":"Python","stars":33,"topics":[],"license":"Apache-2.0","category":"mcp-servers","readme_excerpt":"mcp-eval mcp-eval is an evaluation framework for testing Model Context Protocol (MCP) servers and the agents that use them. Unlike traditional testing approaches that mock interactions or test components in isolation, mcp-eval exercises your complete system in the environment it actually runs in: an LLM/agent calling real MCP tools. Why mcp-eval exists The challenge As AI agents become more sophisticated and MCP servers proliferate, teams face critical questions: For MCP server developers : \"Will my server handle real agent requests correctly? What about edge cases?\" For agent developers : \"Is my agent using tools effectively? Does it recover from errors?\" For both : \"How do we measure quality, performance, and reliability before production?\" The solution mcp-eval addresses these challenges by providing: 1. Real environment testing - No mocks, actual agent-to-server communication 2. Full observability - OpenTelemetry traces capture detailed agent execution to run evals over 3. Rich assertion library - From tool checks to sophisticated path analysis 4. Multiple test styles - Choose what fits your workflow -- pytest, datasets or @task decorators 5. Language agnostic - Test MCP servers written in any language How it works Model Context Protocol standardizes how applications provide context to large language models (LLMs). Think of MCP like a USB-C port for AI applications. mcp-eval ensures your MCP servers, and agents built with them, work reliably in production. Test any MCP se","default_branch":null,"files":null,"tree":[],"storefront":"/r/lastmile-ai","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/lastmile-ai/mcp-eval/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}