{"repo":"llnl/merlin","free":true,"listed":false,"github":"https://github.com/llnl/merlin","clone":"git clone https://github.com/llnl/merlin.git","description":"Machine Learning for HPC Workflows","language":"Python","stars":151,"topics":["machine-learning","big-data","simulation","workflow","redis-server","celery-workers","workflows","hpc","radiuss"],"license":null,"category":"workflow-automation","readme_excerpt":"A brief introduction to Merlin Merlin is a tool for running machine learning based workflows. The goal of Merlin is to make it easy to build, run, and process the kinds of large scale HPC workflows needed for cognitive simulation. At its heart, Merlin is a distributed task queuing system, designed to allow complex HPC workflows to scale to large numbers of simulations (we've done 100 Million on the Sierra Supercomputer). Why would you want to run that many simulations? To become your own Big Data generator. Data sets of this size can be large enough to train deep neural networks that can mimic your HPC application, to be used for such things as design optimization, uncertainty quantification and statistical experimental inference. Merlin's been used to study inertial confinement fusion, extreme ultraviolet light generation, structural mechanics and atomic physics, to name a few. How does it work? In essence, Merlin coordinates complex workflows through a persistent external queue server that lives outside of your HPC systems, but that can talk to nodes on your cluster(s). As jobs spin up across your ecosystem, workers on those allocations pull work from a central server, which coordinates the task dependencies for your workflow. Since this coordination is done via direct connections to the workers (i.e. not through a file system), your workflow can scale to very large numbers of workers, which means a very large number of simulations with very little overhead. Furthermore, si","default_branch":null,"files":null,"tree":[],"storefront":"/r/llnl","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/llnl/merlin/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}