{"repo":"zjunlp/WorfBench","free":true,"listed":false,"github":"https://github.com/zjunlp/WorfBench","clone":"git clone https://github.com/zjunlp/WorfBench.git","description":"[ICLR 2025] Benchmarking Agentic Workflow Generation","language":"Python","stars":157,"topics":["agent","agent-planning","agents","artificial-intelligence","large-language-models","natural-language-processing","workflow","agentic","agentic-workflow","benchmark"],"license":"MIT","category":"workflow-automation","readme_excerpt":"WorfBench Benchmarking Agentic Workflow Generation 📄arXiv • 🤗HFPaper • 🌐Web • 📊Dataset • 🎧NotebookLM Audio Table of Contents - 🌻Acknowledgement - 🌟Overview - 🔧Installation - ✏️Model-Inference - 📝Workflow-Generation - 🤔Workflow-Evaluation - 🚩Citation --- 🌻Acknowledgement Our code of training module is referenced and adapted from LLaMA-Factory. And the Dataset is collected from ToolBench, ToolAlpaca, Lumos, WikiHow, Seal-Tools, Alfworld, Webshop, IntercodeSql. Our end-to-end evaluation module is based on IPR, Stable ToolBench. Thanks for their great contributions! 🌟Overview Large Language Models (LLMs), with their exceptional ability to handle a wide range of tasks, have driven significant advancements in tackling reasoning and planning tasks, wherein decomposing complex problems into executable workflows is a crucial step in this process. Existing workflow evaluation frameworks either focus solely on holistic performance or suffer from limitations such as restricted scenario coverage, simplistic workflow structures, and lax evaluation standards. To this end, we introduce WorFBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures. Additionally, we present WorFEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms to accurately quantify the LLM agent's workflow generation capabilities. Through comprehensive evaluations across different types of LLMs, we discover distinc","default_branch":null,"files":null,"tree":[],"storefront":"/r/zjunlp","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/zjunlp/WorfBench/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}