{"repo":"moritzkoerber/local-data-stack","free":true,"listed":false,"github":"https://github.com/moritzkoerber/local-data-stack","clone":"git clone https://github.com/moritzkoerber/local-data-stack.git","description":"This project demos how to integrate DuckDB and Delta Lake with dbt, orchestrated by Dagster. All data is stored and processed locally.","language":"Python","stars":14,"topics":["dbt","sql","uv","dagster","polars","delta-rs","delta-lake"],"license":null,"category":"data-pipelines","readme_excerpt":"Local data stack This project demos how to integrate DuckDB and Delta Lake with dbt, orchestrated by Dagster. All data is stored and processed locally/on-premise. The whole pipeline is partitioned by date range and scheduled to run daily. The stack queries stock exchange (Xetra) data through an API and upserts the result into a Delta table (= bronze layer). The table serves as a source for dbt, which does a layered incremental load into a DuckDB database: first into silver, then into gold. Finally, the gold table is queried with DuckDB to create a line chart in Plotly. Overview: How to run The stack uses uv to manage its dependencies. Simply run uv run dg dev to launch the dagster webserver.","default_branch":null,"files":null,"tree":[],"storefront":"/r/moritzkoerber","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/moritzkoerber/local-data-stack/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}