{"repo":"GregoryKogan/yt-framework","free":true,"listed":false,"github":"https://github.com/GregoryKogan/yt-framework","clone":"git clone https://github.com/GregoryKogan/yt-framework.git","description":"Build scalable data pipelines on YTsaurus with automatic stage management, local development simulation, and more.","language":"Python","stars":28,"topics":["big-data","data-pipeline","distributed-computing","etl","framework","map-reduce","python","yt","ytsaurus"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"YT Framework PyPI Docs DeepWiki Examples --- Overview Python helpers and conventions for YTsaurus pipelines: YAML config, ordered stages under stages/ , dev mode that mirrors many prod behaviors on disk, and prod mode that uploads src/ bundles to the cluster. Architecture - Pipeline — loads config, builds the YT client, walks enabled stages . - Stage — one BaseStage subclass plus config.yaml (and optional src/ for jobs). - Operations — map, vanilla, map-reduce/reduce, YQL via the client, S3 helpers, sorts, etc. - Configuration — OmegaConf-backed YAML; secrets in configs/secrets.env . What ships in the box - Stage discovery ( DefaultPipeline ) from the filesystem layout. - dev / prod switch on the same code paths where possible. - Map, vanilla, YQL helpers, S3 listing/download patterns, table helpers, checkpoint upload wiring. - Optional custom Docker images, tokenizer tarballs, and multi-operation stages. Installation For Users Install from PyPI into any Python 3.11+ environment (system Python, a virtualenv, or a Conda env): For Developers and Contributors Recommended: one Conda environment for tests, formatting, pre-commit, and local documentation builds (avoids reinstalling tooling for each task): Use conda-forge as the channel when creating the env if that matches your setup ( conda create -n yt-framework python=3.11 -c conda-forge ). Alternative: pip only — install in editable mode from source: For development with testing tools (without the docs extra): For local Sphinx ","default_branch":null,"files":null,"tree":[],"storefront":"/r/GregoryKogan","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/GregoryKogan/yt-framework/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}