{"repo":"godstime-dev/crypto-data-pipeline","free":true,"listed":false,"github":"https://github.com/godstime-dev/crypto-data-pipeline","clone":"git clone https://github.com/godstime-dev/crypto-data-pipeline.git","description":"Real-time crypto data pipeline built with Python, featuring automated ETL workflows, API ingestion, Duckdb storage, market trend analysis, and Discord alerting.","language":"Python","stars":13,"topics":["apis","apscheduler","automation","backend","crypto","data-engineering","data-pipeline","etl","logging","python"],"license":null,"category":"data-pipelines","readme_excerpt":"Crypto ETL Pipeline System A real-time automated data engineering pipeline that collects cryptocurrency prices, processes and analyzes market movements, stores historical data, and triggers intelligent alerts via Discord. This project demonstrates a production-style ETL architecture with scheduling, validation, persistence, and observability, built entirely in Python. --- Features - Automated ETL pipeline (Extract → Transform → Load) - Real-time cryptocurrency data ingestion (CoinGecko API) - Data validation layer to ensure API integrity - Market intelligence engine (trend, volatility, momentum analysis) - Persistent storage using SQLite - Smart alert system with cooldown protection - Discord webhook notifications - Scheduled execution every 5 minutes - Retry logic with exponential backoff - Structured logging system --- System Architecture Collector → Validator → Transformer → Trends Engine → Database → Alert System ↓ Scheduler (runs every 5 min) The system is designed as a modular ETL pipeline where each component is independent, testable, and replaceable. --- Tech Stack - Python 3.x - DuckDB (local persistence) - APScheduler (task scheduling) - Requests (API communication) - Discord Webhooks (notifications) - Standard Library (threading, logging, datetime) --- How It Works 1. Data Collection Fetches Bitcoin and Ethereum prices from CoinGecko API with retry and rate-limit handling. 2. Validation Layer Ensures API responses are structurally valid before processing. 3. Transf","default_branch":null,"files":null,"tree":[],"storefront":"/r/godstime-dev","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/godstime-dev/crypto-data-pipeline/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}