{"repo":"data-burst/airflow-git-sync","free":true,"listed":false,"github":"https://github.com/data-burst/airflow-git-sync","clone":"git clone https://github.com/data-burst/airflow-git-sync.git","description":"Sync DAG changes from Git to Airflow","language":null,"stars":74,"topics":["airflow","airflow-dags","airflow-docker","docker","docker-compose","gitops"],"license":"MIT","category":"deployment-docker-iac","readme_excerpt":"Airflow Git Sync 🎉 What if you could instantly sync DAG changes from Git to Airflow ? Well now you can! Table of Contents 🏗️ - Airflow Git Sync - Table of Contents 🏗️ - Project Description 🌱 - Project Usage 🧑‍💻 - Setting Up Your SSH Key - Using Airflow Git Sync - FAQ ❓ - Contributing 👥 - License 📄 Project Description 🌱 Airflow Git Sync provides automated DAG deployments from Git for Airflow environments. It syncs your pipeline code from a Git repository into the Airflow DAG folder. Keeping DAGs directly in Airflow servers makes management challenging. Code changes require manual syncing to containers. There is no version control or history. If you have ever worked with Airflow on Kubernetes, it gives you the ability to sync the DAGs with your repository (as an GitOps solution) using git-sync sidecar contanier. If you don't have Kubernetes, it is hard to keep the DAGs directory of Airflow (which is placed at /opt/airflow/dags/ ) synced with the changes you applied to your DAGs and in some cases it is required to restart the Airflow service or container. The project introduces git-sync application alongside Airflow. This handles cloning your configured DAG Git repository and syncing contents over to Airflow's DAG directory. The syncing is achieved via a lightweight Docker container that runs periodically using inotify wait to detect file changes. The container can be deployed using docker-compose alongside Airflow. Here is a bit of the docker-compose file: The git-sync","default_branch":null,"files":null,"tree":[],"storefront":"/r/data-burst","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/data-burst/airflow-git-sync/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}