{"repo":"l-mds/local-data-stack","free":true,"listed":false,"github":"https://github.com/l-mds/local-data-stack","clone":"git clone https://github.com/l-mds/local-data-stack.git","description":"Slow & local data allows you to move fast and deliver business value for the 99.9% of the data challenges.","language":"Python","stars":397,"topics":["age","cruft","dagster","dbt","docker","duckdb","pixi","pyright","ruff","sops"],"license":null,"category":"data-pipelines","readme_excerpt":"local data stack template This template brings together some of the best open-source tools in data engineering and Rust-based tooling for ensuring code quality: - dagster, dbt, duckdb - pixi - docker - ruff, pyright, taplo, pytest - cruft While the stack runs locally, it's designed with scale in mind. Data and state accumulate over time (as data engineers say in partitions). Slow data - similar to slow food, is healthy and allows the fast delivery of business value efficiently. The idea of slow data refers to: - focusing on the reduction of complexity - allows to reduce complex data pipelines down to a single developers notebook - making complex data pipeliens testable - shipping real busines value fast - scaling as needed via partitions and a scalable resource provider The template directly can be deployed via docker-compose. With only minimal adaptations you can instead deploy to kubernetes in any cloud provider. Additionally included for secure handling of secrets: - sops - age usage Prerequisites: - git - an installation of cruft pip install cruft jinja2-ospath - an installation of pixi https://pixi.sh/latest/#installation curl -fsSL https://pixi.sh/install.sh bash - text editor of choice such as vscode - docker Post install: - update the secrets in the .env files by executing: openssl rand -base64 32 and setting a suitable secret - ensure the .env.enc can be created by following the instructions in [documentation/secops] Developing the template To render an instance of t","default_branch":null,"files":null,"tree":[],"storefront":"/r/l-mds","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/l-mds/local-data-stack/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}