{"repo":"edgBR/delta-lake-polars","free":true,"listed":false,"github":"https://github.com/edgBR/delta-lake-polars","clone":"git clone https://github.com/edgBR/delta-lake-polars.git","description":"Building a poor man's data lake: Exploring the Power of Polars and Delta Lake","language":"Python","stars":11,"topics":["data-engineering","datalake","delta","delta-lake","polars","polars-dataframe"],"license":null,"category":"data-pipelines","readme_excerpt":"Introduction This repository contains examples of using Polars and Deltalake directly against ADLSGen2 Storage. For maximum reproducibility they use the base interfaces of azure storage and minimal dependencies are added (no AzureML Data). The repository uses poetry to manages dependencies instead of pip. Poetry offers a much richer configuration environment files via pyproject.tml and poetry.lock. Unlike pip, poetry.lock locks the dependencies at binary level and in a hasheable format, ensuring even better reproducibility. Besides it allow us to combine nicely with pre-commit hooks and linting tools like ruff or pylint. Getting Started To get started you should perform the following steps: 1. Installation of poetry: - Ensure you have a python installation (higher than 3.8+) in your system (via miniconda3, anaconda3 or a vanilla one from python.org) - Run the following command in a linux computer or in WSL (Windows Subsystem for Linux): - This will install poetry in the current python environment that you have active. Poetry needs to be separated always from your project environment. If you are using conda or miniconda for your python installation you can install poetry in your base environment. However please ensure that your base environment is clean and only contains minimal dependencies. 2. Configuring poetry: - Sometimes (specially if you have conda) poetry PATH variable can be overwritten when you run . - Sometimes your system will not recognize the path of poetry (this","default_branch":null,"files":null,"tree":[],"storefront":"/r/edgBR","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/edgBR/delta-lake-polars/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}