{"repo":"google/litmus","free":true,"listed":false,"github":"https://github.com/google/litmus","clone":"git clone https://github.com/google/litmus.git","description":"Litmus is a comprehensive LLM testing and evaluation tool designed for GenAI Application Development. It provides a robust platform with a user-friendly UI for streamlining the process of building and assessing the performance of your LLM-powered applications.","language":"Vue","stars":51,"topics":["api","apitesting","cicd","devops","llm","llm-evaluation","llm-security","llm-training","llmops","testing"],"license":"Apache-2.0","category":"deployment-docker-iac","readme_excerpt":"Litmus: A comprehensive LLM testing and evaluation tool designed for GenAI Application Development. Litmus is a comprehensive tool designed for testing and evaluating HTTP Requests and Responses, especially for Large Language Models (LLMs). It combines a powerful API, a robust worker service, a user-friendly web interface, and an optional proxy service to streamline the testing process. Features - Automated Test Execution: Submit test runs using pre-defined templates to evaluate responses against golden answers using AI. - Flexible Test Templates: Define and manage test templates specifying the structure and parameters of your tests. You can choose between two template types: \"Test Run\" for single-turn interactions and \"Test Mission\" for multi-turn interactions where the LLM generates its requests. - User-Friendly Web Interface: Interact with the Litmus platform through a visually appealing and intuitive web interface. - Detailed Results: View the status, progress, and detailed results of your test runs. - Advanced Filtering: Filter responses from test runs based on specific JSON paths for in-depth analysis. - Performance Monitoring: Track the performance of your responses and identify areas for improvement by using AI. - Multiple LLM Evaluation Methods: Leverage a variety of LLM evaluation methods: - Custom LLM Evaluation with Customizable Prompts: Use an LLM to compare actual responses with expected (golden) responses, utilizing flexible prompts tailored to your evaluation ","default_branch":null,"files":null,"tree":[],"storefront":"/r/google","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/google/litmus/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}