{"repo":"josephmachado/cost_effective_data_pipelines","free":true,"listed":false,"github":"https://github.com/josephmachado/cost_effective_data_pipelines","clone":"git clone https://github.com/josephmachado/cost_effective_data_pipelines.git","description":"Cost Efficient Data Pipelines with DuckDB","language":"C","stars":62,"topics":["duckdb","python","sql"],"license":null,"category":"databases-storage","readme_excerpt":"Cost Effective Data Pipelines Setup Run on CodeSpaces Run code on your machine Generate data Data processing Cost Effective Data Pipelines Code for blog at: Building cost effective data pipelines with Python & DuckDB Setup Run on CodeSpaces You can run the code in the repo with Codespaces. 1. Fork this repo. 2. Click on code button and select the codespaces tab. 3. Click on the Create codespace on main button. When codespaces open, wait for it to complete installing the required libraries from ./requirements.txt. You can open a Python REPL and run the scripts in the generate data section. Run code on your machine Alternatively you can run this code on your machine. You'll need the following: 1. Python 3.8 or above 2. sqlite3 3. Sufficient disk memory (depending on if you want to run with 1 or 10 or 100GB) Clone the repo and create a virtual env and install the libraries needed: Generate data For the example in this repo we use the TPC-H data set and Coincap API. Let's generate the TPCH data, by running the following commands in your terminal: Let's open a sqlite3 shell and run a quick count check to ensure that the tables were loaded properly. Data processing You can run the ETL scripts using the make up command. Alternatively you can run the scripts individually as shown below. Running the ETL on a 8 core, 32 GB RAM, 1TB HDD 2017 Thinkpad with performs as shown below. The script wide month supplier metrics.py ran in 7m and 20s, this included reading in about 10GB of data, in","default_branch":null,"files":null,"tree":[],"storefront":"/r/josephmachado","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/josephmachado/cost_effective_data_pipelines/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}