{"repo":"damavis/airflow-pentaho-plugin","free":true,"listed":false,"github":"https://github.com/damavis/airflow-pentaho-plugin","clone":"git clone https://github.com/damavis/airflow-pentaho-plugin.git","description":"Pentaho plugin for Apache Airflow - Orquestate pentaho transformations and jobs from Airflow","language":"Python","stars":40,"topics":["airflow-plugin","airflow","data-engineering","pentaho-data-integration"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Pentaho Airflow plugin This plugins runs Jobs and Transformations through Carte servers. It allows to orchestrate a massive number of trans/jobs taking care of the dependencies between them, even between different instances. This is done by using CarteJobOperator and CarteTransOperator It also runs Pan (transformations) and Kitchen (Jobs) in local mode, both from repository and local XML files. For this approach, use KitchenOperator and PanOperator Requirements 1. A Apache Airflow system deployed. 2. One or many working PDI CE installations. 3. A Carte server for Carte Operators. Setup The same setup process must be performed on webserver, scheduler and workers (that runs this tasks) to get it working. If you want to deploy specific workers to run this kind of tasks, see Queues, in Airflow Concepts section. Pip package First of all, the package should be installed via pip install command. Airflow connection Then, a new connection needs to be added to Airflow Connections, to do this, go to Airflow web UI, and click on Admin - Connections on the top menu. Now, click on Create tab. Use HTTP connection type. Enter the Conn Id , this plugin uses pdi default by default, the username and the password for your Pentaho Repository. At the bottom of the form, fill the Extra field with pentaho home , the path where your pdi-ce is placed, and rep , the repository name for this connection, using a json formatted string like it follows. Carte In order to use CarteJobOperator , the connectio","default_branch":null,"files":null,"tree":[],"storefront":"/r/damavis","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/damavis/airflow-pentaho-plugin/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}