{"repo":"ebonnal/streamable","free":true,"listed":false,"github":"https://github.com/ebonnal/streamable","clone":"git clone https://github.com/ebonnal/streamable.git","description":"sync/async iterable streams for Python","language":"Python","stars":331,"topics":["data-engineering","etl-pipeline","etl","iterable","reverse-etl","collections","streams","fluent-interface","iterator-pattern","lazy-evaluation"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"༄ streamable sync/async iterable streams for Python stream[T] wraps any Iterable[T] or AsyncIterable[T] with a lazy fluent interface covering concurrency, batching, buffering, rate limiting, progress observation, and error handling. 1. install (no dependencies) 2. import 3. init Create a stream[T] from an Iterable[T] or AsyncIterable[T] : 4. operate Chain lazy operations (accepting sync and async functions): Source elements will be processed on-the-fly during iteration. 5. iterate A stream[T] is Iterable[T] and AsyncIterable[T] : --- 📖 operations (docs) - mapping - .map - .do - (un)grouping - .group - .flatten - filtering - .filter - .take - .skip - control - .catch - .throttle - .buffer - .observe A stream exposes operations to manipulate its elements, but its consumption and the I/O are not its responsibility. It's meant to be combined with dedicated libraries like pyarrow , psycopg2 , dlt (ETL example) ... A stream can be iterated several times if its source allows it. Operations return a new stream . Operations allow iteration to resume after an exception. Operations accept both sync and async functions. A sync iteration over a stream involving async functions runs an async iteration under the hood, using the current event loop (or setting a new one if needed). Operations adapt their behavior to the type of iteration. see examples - .throttle sleeps via time.sleep during sync iteration and via asyncio.sleep during async iteration. - .buffer 's background task is a thread","default_branch":null,"files":null,"tree":[],"storefront":"/r/ebonnal","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ebonnal/streamable/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}