{"repo":"arthurprevot/yaetos","free":true,"listed":false,"github":"https://github.com/arthurprevot/yaetos","clone":"git clone https://github.com/arthurprevot/yaetos.git","description":"Write data & AI pipelines in (SQL, Spark, Pandas) and deploy to the cloud, simplified","language":"Python","stars":36,"topics":["etl","data-engineering","aws","spark","big-data","small-data","pandas","machine-learning"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Yaetos Yaetos is a framework to write data pipelines on top of Pandas and Spark, and deploy them to Amazon Web Services (AWS). It can run locally or on AWS (using S3 datasets and running the process on an AWS cluster). The focus is on making simple things easy and complex things possible (and as easy as can be). It's name stands for \"Yet Another ETL Tool on Spark\". - In the simplest cases, pipelines consist of SQL files only. No need to know any programming. Suitable for business intelligence use cases. - In more complex cases, pipelines consist of python files, giving access to Pandas, Spark dataframes, RDDs and any python library (scikit-learn, tensorflow, pytorch). Suitable for AI use cases. It integrates several popular open source systems:                Some features: The ability to run jobs locally and on a cluster in the cloud without any changes. The support for dependencies across jobs The support for incremental jobs (inc. idempotency) The automatic creation of AWS clusters when needed. The support for git and unit-tests The ability to integrate any python library in the process (ex: machine learning libraries). Getting Started https://user-images.githubusercontent.com/3277100/175531451-1931086d-866a-40a8-8b1d-0417f8928b66.mp4 Run the commands from the \"installation instructions\" section below. Then run this sql example locally with: yaetos run dockerized jobs/generic/launcher.py --job name=examples/ex1 sq","default_branch":null,"files":null,"tree":[],"storefront":"/r/arthurprevot","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/arthurprevot/yaetos/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}