{"repo":"sophie-nguyenthuthuy/data-engineering","free":true,"listed":false,"github":"https://github.com/sophie-nguyenthuthuy/data-engineering","clone":"git clone https://github.com/sophie-nguyenthuthuy/data-engineering.git","description":"100+ data engineering projects from scratch — streaming, CDC, table formats, query engines, consensus, governance. 2,500+ tests, mypy strict.","language":"Python","stars":53,"topics":["cdc","data-engineering","delta-lake","iceberg","kafka","lsm-tree","python","query-optimizer","raft","streaming"],"license":null,"category":"data-pipelines","readme_excerpt":"🛠️ Data Engineering — 100+ Production-Grade Projects One of the most comprehensive open-source data-engineering portfolios on GitHub. From-scratch implementations of the systems behind every modern data platform — Kafka • Iceberg • Delta Lake • Hudi • Parquet • CDC • Vector Clocks • CRDTs • LSM Trees • Differential Privacy • Causal Consistency • Bayesian DQ. ⭐ If this repo is useful, please give it a star — it's how I know what to build next. --- [!TIP] TL;DR — 100+ self-contained data-engineering projects. Real algorithms, 2 500+ passing tests , mypy --strict , multi-stage Dockerfiles, GitHub Actions matrix on Python 3.10/3.11/3.12, MIT-licensed. Zero or minimal runtime dependencies. Browse the directory ↓. A curated monorepo of data-engineering work — pipelines, storage engines, query engines, table formats, streaming, governance, and distributed-systems primitives. Each subdirectory is an independent project with its own README, tests, Docker image, and CI workflow. ✨ Why this repo - 🧱 Production-grade , not toy examples. Every project ships with strict type-checking, ruff lint, hand-written tests, Docker images, and GHA CI. - 🧬 From-scratch implementations of the algorithms that power Kafka / Iceberg / Delta / Hudi / Parquet / CDC / Postgres WAL — read the code, learn how they actually work. - 🐍 Zero or minimal runtime dependencies — most projects use only the Python stdlib. The wire formats are real; the code is auditable. - ⚡ One-command local runs — cd && pip insta","default_branch":null,"files":null,"tree":[],"storefront":"/r/sophie-nguyenthuthuy","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/sophie-nguyenthuthuy/data-engineering/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}