{"repo":"It4innovations/HyperLoom","free":true,"listed":false,"github":"https://github.com/It4innovations/HyperLoom","clone":"git clone https://github.com/It4innovations/HyperLoom.git","description":"Platform for defining and executing scientific pipelines in distributed environments","language":"C++","stars":19,"topics":["distributed-systems","hpc","workflow","pipelines","scheduler","task"],"license":null,"category":"workflow-automation","readme_excerpt":"HyperLoom HyperLoom is a platform for defining and executing workflow pipelines in large-scale distributed environments. HyperLoom implements its own schedulling algorithm optimized for execution of millions of interconnected tasks on hundreds of computational nodes. HyperLoom also includes a thin Python client module that allows to easily define and execute the pipelines on HyperLoom infrastructure. HyperLoom features: In-memory data processing reducing filesystem load. Direct worker-to-worker data transfer reducing server overhead. Support for execution of third party applications. Data-location aware scheduling algorithm reducing inter-node network traffic. C++ core with a Python client enabling high performance through a simple API. High scalability and native HPC support. BSD license. For more information see the full documentation. Architecture Quickstart Execute your first HyperLoom pipeline in 4 easy steps using Docker: 1. Deploy virtualized HyperLoom infrastructure Note that before re-running docker-compose up you need to run docker-compose down to delete containers state. 2. Install HyperLoom client (virtualenv) 3. Define a pipeline Create a python file pipeline.py with the following content: 4. Execute the pipeline Documentation The compiled version of the documentation is available here. You can also build the full documentation from the sources in the doc subdirectory by running make html . Citations Please use the following BibTeX record when refering to HyperLo","default_branch":null,"files":null,"tree":[],"storefront":"/r/It4innovations","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/It4innovations/HyperLoom/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}