{"repo":"kevin-hanselman/dud","free":true,"listed":false,"github":"https://github.com/kevin-hanselman/dud","clone":"git clone https://github.com/kevin-hanselman/dud.git","description":"A lightweight CLI tool for versioning data alongside source code and building data pipelines.","language":"Go","stars":220,"topics":["machine-learning","dvcs","data-science","dataset","mlops","data-engineering","data-pipelines"],"license":"BSD-3-Clause","category":"data-pipelines","readme_excerpt":"Dud Website Install Getting Started Source Code Dud is a lightweight tool for versioning data alongside source code and building data pipelines. In practice, Dud extends many of the benefits of source control to large binary data. With Dud, you can commit, checkout, fetch, and push large files and directories with a simple command line interface. Dud stores recipes (a.k.a. stages) for retrieving your data in small YAML files. These stages can be stored in source control to link your data to your code . On top of that, stages can run the commands to generate the data , sort of like Make. Stages can be chained together to create data pipelines . See the Getting Started guide for a hands-on overview. Dud is pronounced \"duhd\", not \"dood\". Dud is not an acronym. Motivation Dud is heavily inspired by DVC. DVC addresses the need for data versioning and reproducibility, but its implementation is not without problems. My criticisms of DVC boil down to two things: speed and simplicity. By speed, I mean throughput and responsiveness. By simplicity, I mean doing less--both in project scope and amount of abstraction. In terms of speed, Dud is generally much faster than DVC. In terms of simplicity, Dud has a smaller, more focused scope, and it is distributed as a standalone executable. To summarize with an analogy: Dud is to DVC what [Flask][1] is to [Django][1]. Both Dud and DVC have their strengths. If you want a \"batteries included\" suite of tools for managing machine learning projects,","default_branch":null,"files":null,"tree":[],"storefront":"/r/kevin-hanselman","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/kevin-hanselman/dud/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}