{"repo":"mehd-io/pypi-duck-flow","free":true,"listed":false,"github":"https://github.com/mehd-io/pypi-duck-flow","clone":"git clone https://github.com/mehd-io/pypi-duck-flow.git","description":"end-to-end data engineering project to get insights from PyPi using python, duckdb, MotherDuck","language":"TypeScript","stars":237,"topics":["dataengineering","duckdb","etl","python","bigquery","motherduck"],"license":null,"category":"data-pipelines","readme_excerpt":"Pypi Duck Flow : Get insights of your python project 🐍 This project is a collections of pipelines to get insights of your python project. It also serves as educational purpose (YouTube videos and blogs) to learn how to build data pipelines with Python , SQL & DuckDB . You can see the final result of the project in this live dashboard. The project is a monorepo composed of series in 3 parts : - Ingestion, under ingestion folder (YouTube video, Blog) - transformation, under transform folder (YouTube video, Blog) - Visualization, under dashboard folder (YouTube video, Blog) ⚠️ This project has undergone significant changes since the tutorials were recorded. To follow along with the original tutorial content, please check out the feat/tutorial-archive branch. You can also refer to the CHANGELOG.md for a complete list of updates. High level architecture Development Setup The project requires : Python 3.12 uv for python packages. Node.js (only for the visualization part) There's also two devcontainers definitions for VSCode : one for Python, and one for NodeJS. Finally a Makefile is available to run common tasks. Env & credentials A .env file is required to run the project. You can copy env.template and fill the required values. Ingestion The pipeline reads from the public PyPI BigQuery dataset using the DuckDB BigQuery community extension ( bigquery scan with filter pushdown via the Storage Read API) and writes directly to MotherDuck. SQL-based data quality checks (null-rate thre","default_branch":null,"files":null,"tree":[],"storefront":"/r/mehd-io","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mehd-io/pypi-duck-flow/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}