{"repo":"MarkPhamm/local_warehouse_pipeline","free":true,"listed":false,"github":"https://github.com/MarkPhamm/local_warehouse_pipeline","clone":"git clone https://github.com/MarkPhamm/local_warehouse_pipeline.git","description":"A local-first Data Warehouse ELT pipeline that extracts CFPB consumer complaint data, transforms it with dbt into analytics-ready models, and serves interactive dashboards—all running on your laptop with zero cloud dependencies.","language":"Jupyter Notebook","stars":11,"topics":["dlt","duckdb","git","github","prefect","visivo","visivo-dashboard","dbt"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Local Data Warehouse A local-first Data Wareouse pipeline that extracts CFPB consumer complaint data, transforms it with dbt into analytics-ready models, and serves interactive dashboards—all running on your laptop with zero cloud dependencies. Package Manager : uv (Python) Ingestion : dlt + PyArrow (API → Parquet staging → DuckDB) Staging Format : Parquet (via PyArrow) in landing/ Transformation & Documentation : dbt core & dbt-colibri OLAP Database : DuckDB Orchestration : Prefect BI Tool : Visivo 1. Quick Start 1.1 Setup 1.2 Configuration Edit src/cfg/config.py to configure companies and start date: 1.3 Run the Pipeline 1.4 Backfill Landing Area The landing area stores daily parquet files under landing/cfpb complaints/YYYY MM DD/ , one file per company per day. Use the backfill script to populate historical data: Each daily directory contains 10 parquet files (one per company). Empty parquet files are created for days with no complaints to maintain a consistent structure. 1.5 Access Prefect UI (Optional) 1.6 Access DuckDB UI First, install the DuckDB CLI: Then launch the DuckDB UI: We need to add in our database path as follow: Additional docs: DuckDB UI Documentation 1.7 Access Visivo Dashboards Start the Visivo web server to view interactive dashboards: Then open your browser to: Dashboard URL : Available Dashboards : Executive Dashboard : High-level overview of complaint trends and company performance Geographic Dashboard : State-by-state complaint distribution and anal","default_branch":null,"files":null,"tree":[],"storefront":"/r/MarkPhamm","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/MarkPhamm/local_warehouse_pipeline/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}