{"repo":"fortiql/data-forge","free":true,"listed":false,"github":"https://github.com/fortiql/data-forge","clone":"git clone https://github.com/fortiql/data-forge.git","description":"Data Forge — a modern data stack playground to practice flows and best practices, not just tools. Spark, Trino, Kafka, Iceberg, ClickHouse, Airflow, MinIO, Superset — all wired together locally with Docker Compose.","language":"Jupyter Notebook","stars":182,"topics":["bronze-silver-gold","data-engineering","data-lake","data-simulation","lakehouse","modern-data-stack","open-source"],"license":null,"category":"data-pipelines","readme_excerpt":"🔥 Data Forge — Data Engineering Playground Your modern data stack playground. Spin up core components of a real data platform and practice end‑to‑end workflows locally. Instead of just reading about \"data lakes\" or \"lakehouses,\" you actually get to run them. Think of it as a gym for data engineers without cloud bills or production risk. --- 🎯 What's Inside Data Forge includes a complete modern data stack with industry-standard tools: 🗄️ Storage & Catalog - MinIO → S3-compatible object storage for data lakes - Hive Metastore → Centralized metadata catalog for tables and schemas ⚡ Compute Engines - Trino → Interactive SQL query engine for federated analytics - Apache Spark → Distributed processing for batch and streaming workloads 🌊 Streaming & CDC - Apache Kafka → Event streaming platform - Schema Registry → Schema evolution and compatibility - Debezium → Change data capture from databases 🗃️ Databases - PostgreSQL → Primary OLTP database (source system) - ClickHouse → Columnar analytics database (sink) 🔄 Orchestration - Apache Airflow 3 → Workflow orchestration 📊 Visualization & Exploration - Apache Superset → Modern BI and data visualization - JupyterLab → Interactive data science environment 🏭 Data Generation - Data Generator → Realistic retail data producer for Kafka topics and Postgres tables (see infra/data-generator/README.md) --- 🚀 Quick Start Prerequisites - Docker 20.10+ - Docker Compose 2.0+ - 8GB+ RAM recommended - 20GB+ disk space for all services 1. Clon","default_branch":null,"files":null,"tree":[],"storefront":"/r/fortiql","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/fortiql/data-forge/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}