{"repo":"Point72/polars-io-tools","free":true,"listed":false,"github":"https://github.com/Point72/polars-io-tools","clone":"git clone https://github.com/Point72/polars-io-tools.git","description":"Custom parsing extensions for lazy polars","language":"Python","stars":18,"topics":["clickhouse","lazy","lazy-evaluation","polars","python","rust","sql"],"license":"Apache-2.0","category":"databases-storage","readme_excerpt":"polars io tools Custom parsing extensions for lazy polars Overview polars-io-tools extends Polars lazy execution with custom I/O sources that push filters and column projections all the way down into the systems that hold your data — SQL databases, ClickHouse, Datadog, and Delta Lake — instead of loading everything and filtering in memory. It also adds lazy-friendly operations (joins, multi-source composition, time-series windows, caching, distributed execution) that keep predicate pushdown working where vanilla Polars would otherwise give up and materialize the whole frame. Everything is exposed through the piot LazyFrame namespace and a handful of top-level scan / sink functions, so it composes naturally with the Polars API you already use. Who is this for Reach for polars-io-tools when you want Polars' lazy API over data that lives in an external store, and you care about not fetching rows or columns you will immediately throw away. It is most valuable for large, partitioned, or remote datasets where a filter on a date or key column should translate into a smaller query against the source. If your data already fits comfortably in memory or lives in local Parquet/CSV, plain Polars is the simpler choice. Installation polars-io-tools requires Python 3.11 or newer. See the Installation guide for conda and source builds. Quickstart Importing the package registers the piot namespace on every Polars LazyFrame : For a guided walkthrough, start with the Getting Started tutorial. Wh","default_branch":null,"files":null,"tree":[],"storefront":"/r/Point72","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Point72/polars-io-tools/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}