{"repo":"mattiasthalen/obsidian-insights","free":true,"listed":false,"github":"https://github.com/mattiasthalen/obsidian-insights","clone":"git clone https://github.com/mattiasthalen/obsidian-insights.git","description":"Personal project for setting up an open source data warehouse.","language":"Python","stars":32,"topics":["data-warehouse","dlt","duckdb","elt","motherduck","sqlmesh","unified-star-schema"],"license":"LGPL-3.0","category":"data-pipelines","readme_excerpt":"Obsidian Insights Goals: - [x] Extract & load Northwind via REST API to MotherDuck using dlt. - [x] Transform using SQLMesh. - [x] Model the silver layer according to The Hook Cookbook. - [x] Model the gold layer as a Unified Star Schema. - [x] Add GitHub Actions for CI/CD and daily ELT. - [ ] Add Apache Iceberg as an alternative to MotherDuck. How To Run 1. Clone the repo. 1. Run pip install uv , followed by uv sync . 1. Decide if you want to use DuckDB (local) or MotherDuck (remote) as gateway. DuckDB (default) 1. Add the gateway var in .env: gateway=duckdb . (fallback is this) 1. Add the duckdb path var in .env: duckdb path=your db.duckdb . (default is ./obsidian insights.duckdb ) MotherDuck 1. Add the gateway var in .env: gateway=motherduck . (fallback is duckdb ) 1. Create an account on MotherDuck. 1. Create a database called obsidian insights . 1. Create a token. 1. Add the motherduck token var in .env: motherduck token=your token . 1. Run init warehouse.sh and follow the prompts. 1. Then run elt.sh whenever you want to refresh the warehouse. Architecture Formulas !NOTE I'm using this definition of what a measure is: A measure is a raw quantifiable value representing a specific aspect of performance, status, or characteristics that must include a temporal anchor specifying the exact point or period in time to which it refers. I.e., a measure must be associated with a date. E.g., the amount on an invoice is associated with three dates; incoive date, due date, and payment","default_branch":null,"files":null,"tree":[],"storefront":"/r/mattiasthalen","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mattiasthalen/obsidian-insights/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}