{"repo":"ORNL/flowcept","free":true,"listed":false,"github":"https://github.com/ORNL/flowcept","clone":"git clone https://github.com/ORNL/flowcept.git","description":"Runtime provenance for AI and scientific workflows—capture, enrich, and query workflow data via observability adapters and code annotation across edge, cloud, and HPC.","language":"Python","stars":32,"topics":["big-data","data-integration","lineage","machine-learning","model-management","parallel-processing","provenance","scientific-workflows","workflows","reproducibility"],"license":"MIT","category":"machine-learning","readme_excerpt":"Runtime provenance for distributed scientific and AI workflows Capture, stream, query, visualize, and reason over workflow lineage across ML, agentic, edge, cloud, and HPC systems. Website · Documentation · Examples · Notebooks · Publications What Is Flowcept? Flowcept is a lightweight runtime provenance data management and data observability system for AI, scientific, and agentic workflows. It records what ran, what data were used and generated, where it ran, how long it took, which resources it consumed, and how workflow artifacts relate to each other, with rich context and detailed metadata. It is designed for workflows that are distributed, heterogeneous, and hard to inspect after the fact: ML training, agentic workflows, HPC jobs, edge-to-cloud-to-HPC pipelines, Workflow Management System tasks, and multi-workflow campaigns. Flowcept records can start as simple JSONL files for small local demos, move to LMDB for pure file-based runs, and scale to MongoDB for cloud deployments, production-style querying, dashboards, and UI use. For distributed runs, provenance records can also stream through a message queue while the workflow is still executing. The Flowcept Agent adds LLM-based interaction on top of captured provenance, so users can ask questions, inspect lineage, and drive visual exploration naturally, both to the in-motion data in the MQ systems or the data persisted in the database. See It In Action The Flowcept UI turns captured provenance into browsable workflow str","default_branch":null,"files":null,"tree":[],"storefront":"/r/ORNL","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ORNL/flowcept/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}