{"repo":"idesis-gmbh/GitHubExperiments","free":true,"listed":false,"github":"https://github.com/idesis-gmbh/GitHubExperiments","clone":"git clone https://github.com/idesis-gmbh/GitHubExperiments.git","description":"Experiments with GitHub event data","language":"Python","stars":20,"topics":["data-warehouse","dbt","dimensional-modeling","duckdb","star-schema","rill"],"license":"MIT","category":"data-pipelines","readme_excerpt":"GitHubExperiments A worked example of building a local data warehouse on the GitHub Archive using dbt-duckdb — from raw JSON to a star schema with slowly changing dimensions, running entirely locally. Blog Posts - From Raw JSON to Data Warehouse: Analyzing GitHub Data Locally - Bringing the Data Warehouse to Life: Exploring GitHub Data Interactively with Rill Getting Started Prerequisites - Python 3.13 or higher - uv — Python package manager - Git - wget or curl for downloading dumps - 3 GB free disk space per day of GitHub Archive data Installation Clone the repository and install dependencies: uv sync reads pyproject.toml and installs all dependencies into a local virtual environment automatically. Data Download Download GitHub Archive data into the data/gharchive/ directory. Using wget : Adjust the date and hour range to your needs, e.g. a full day or month: Data Compressed DuckDB ------ ------------ -------- 1 hour 50 MB 100 MB 1 day 1 GB 2 GB 1 week 7 GB 14 GB 1 month 30 GB 60 GB Running the Pipeline On first run, process the first file and generate dbt models using the canonical sample: Then process all remaining files incrementally: Each file is processed through the full dbt pipeline — staging, snapshots, dimensions, facts, and marts — and acknowledged in the control schema on success. Re-running skips already processed files. The generated SQL models and canonical sample are checked in — --canonical-schema only needs to be rerun after a database reset or if the GitHu","default_branch":null,"files":null,"tree":[],"storefront":"/r/idesis-gmbh","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/idesis-gmbh/GitHubExperiments/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}