{"repo":"godatadriven/whirl","free":true,"listed":false,"github":"https://github.com/godatadriven/whirl","clone":"git clone https://github.com/godatadriven/whirl.git","description":"Fast iterative local development and testing of Apache Airflow workflows","language":"Shell","stars":205,"topics":["airflow","docker","local-development","testing"],"license":"Apache-2.0","category":"deployment-docker-iac","readme_excerpt":"whirl Fast iterative local development and testing of Apache Airflow workflows. The idea of whirl is pretty simple: use Docker containers to start up Apache Airflow and the other components used in your workflow. This gives you a copy of your production environment that runs on your local machine. You can run your DAG locally from start to finish - with the same code as in production. Seeing your pipeline succeed gives you more confidence about the logic you are creating/refactoring and how it integrates with other components. It also gives new developers an isolated environment for experimenting with your workflows. whirl connects the code of your DAG and your (mock) data to the Apache Airflow container that it spins up. Using volume mounts you are able to make changes to your code in your favorite IDE and immediately see the effect in the running Apache Airflow UI on your machine. This also works with custom Python modules that you are developing and using in your DAGs. NOTE: whirl is not intended to replace proper (unit) testing of the logic you are orchestrating with Apache Airflow. Prerequisites whirl relies on Docker and Docker Compose. Make sure you have it installed. If using Docker for Mac or Windows ensure that you have configured it with sufficient RAM (8GB or more recommended) for running all your containers. When you want to use whirl in your CI pipeline, you need to have jq installed. For example, with Homebrew: The current implementation was developed on macOS ","default_branch":null,"files":null,"tree":[],"storefront":"/r/godatadriven","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/godatadriven/whirl/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}