{"repo":"edwinweber/dbt_duckdb_demo_public","free":true,"listed":false,"github":"https://github.com/edwinweber/dbt_duckdb_demo_public","clone":"git clone https://github.com/edwinweber/dbt_duckdb_demo_public.git","description":"Open Source data engineering demo project using dbt, DuckDB, dlt, Dagster and Metabase. Two storage modes for the delta tables are supported: local and Microsoft Fabric Onelake.","language":"Python","stars":47,"topics":["data-engineering","dbt","delta-lake","dlt","duckdb","medallion-architecture","microsoft-fabric","motherduck","open-data","python"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Single-Server Data Engineering: A Reference Implementation Danish Democracy Data is a working data pipeline that runs on a €35/month virtual server (including storage and backups). It ingests open data from two sources (the Danish Parliament OData API and the Rfam public MySQL database), transforms it through a medallion architecture in DuckDB, and optionally exports Delta Lake tables to Microsoft Fabric OneLake. Built with open-source tools — Dagster, dlt, dbt — it's intended as a reference implementation for data engineers who want to understand how these pieces fit together on a single server without cloud lock-in or hidden costs. The Danish Parliament data is the demo; the patterns apply to any stack of heterogeneous sources. This is not a tutorial. It's a working pipeline that handles real data, respects the single-writer constraint of DuckDB, manages incremental and full-extract strategies, tracks data lineage, and observes itself. It is deliberately single-node — no Kubernetes, no high availability, no distributed consensus. The trade-off is simplicity and cost. If you scale past one server, the architecture still applies; you'll just add infrastructure. Quick Start (Docker) With MinIO (local S3 development): To test S3-compatible storage locally using MinIO, use the Docker Compose override file: See docker/DOCKER USAGE.md for details. For local (no Docker) setup, see CLAUDE.md → Running the Project. Documentation Before diving into the code, read these in order: What ","default_branch":null,"files":null,"tree":[],"storefront":"/r/edwinweber","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/edwinweber/dbt_duckdb_demo_public/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}