{"repo":"smanindokhan/crypto-pipeline","free":true,"listed":false,"github":"https://github.com/smanindokhan/crypto-pipeline","clone":"git clone https://github.com/smanindokhan/crypto-pipeline.git","description":"Production data pipeline: CoinGecko API → PostgreSQL → dbt → Airflow orchestration, with 26 automated data quality tests","language":"Python","stars":45,"topics":["airflow","coingecko-api","dbt","dbt-postgres","dbt-utils","docker","metabase","pipeline","python"],"license":null,"category":"data-pipelines","readme_excerpt":"Crypto Market Data Pipeline A production-style data pipeline that ingests live cryptocurrency data daily, transforms it through a layered dbt model, runs automated quality tests, and is orchestrated by Apache Airflow — all running on Docker. --- Architecture Lineage graph Metabase Airflow --- Stack Layer Technology --- --- Ingestion Python 3.12, requests, psycopg2 Storage PostgreSQL 15 (Docker) Transformation dbt-postgres 1.10, dbt-utils Orchestration Apache Airflow 2.10.5 (LocalExecutor, Docker) Visualisation Metabase (Docker) Version control Git + GitHub --- Data sources Source Endpoint Frequency Raw table --- --- --- --- CoinGecko /coins/markets Daily 03:30 UTC raw.coin prices CoinGecko /global Daily 03:30 UTC raw.global market --- dbt model layers Model Layer Materialisation Description --- --- --- --- stg coin prices staging view Cleaned coin price snapshots stg global market staging view Cleaned global market snapshots mart daily prices marts table Daily OHLC + 7-day rolling avg + DoD change mart top movers marts table Top 5 gainers and losers per day mart market dominance marts table BTC/ETH dominance trend with prices --- Data quality - 26 automated dbt tests run after every pipeline execution - not null on all key columns - unique on surrogate keys and date columns - accepted values on categorical columns - Custom singular tests: price always positive, dominance never exceeds 100% - Source freshness check: fails if no new data in 25 hours --- How to run Prerequisites","default_branch":null,"files":null,"tree":[],"storefront":"/r/smanindokhan","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/smanindokhan/crypto-pipeline/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}