{"repo":"gestaogovbr/FastETL","free":true,"listed":false,"github":"https://github.com/gestaogovbr/FastETL","clone":"git clone https://github.com/gestaogovbr/FastETL.git","description":"Airflow plugins for implementing data pipelines. | Plugins do Airflow para implementação de pipelines de dados.","language":"Python","stars":52,"topics":["airflow","airflow-operators"],"license":"GPL-3.0","category":"data-pipelines","readme_excerpt":"FastETL framework, modern, versatile, does almost everything. Este texto também está disponível em português: 🇧🇷LEIAME.md. --- FastETL is a plugins package for Airflow for building data pipelines for a number of common scenarios. Main features: Full or incremental replication of tables in SQL Server, Postgres and MySQL databases Load data from GSheets and from spreadsheets on Samba/Windows networks Extracting CSV from SQL Clean data using custom data patching tasks (e.g. for messy geographical coordinates, mapping canonical values for columns, etc.) Using a Open Street Routing Machine service to calculate route distances Using CKAN or dados.gov.br's API to update dataset metadata Using Frictionless Tabular Data Packages to write OpenDocument Text format data dictionaries This framework is maintained by a network of developers from many teams at the Ministry of Management and Innovation in Public Services and is the cumulative result of using Apache Airflow, a free and open source tool, starting in 2019. For government: FastETL is widely used for replication of data queried via Quartzo (DaaS) from Serpro. Installation in Airflow FastETL implements the standards for Airflow plugins. To install it, simply add the apache-airflow-providers-fastetl package to your Python dependencies in your Airflow environment. Or install it with To see an example of an Apache Airflow container that uses FastETL, check out the airflow2-docker repository. To ensure appropriate results, please mak","default_branch":null,"files":null,"tree":[],"storefront":"/r/gestaogovbr","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/gestaogovbr/FastETL/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}