{"repo":"dsaidgovsg/airflow-pipeline","free":true,"listed":false,"github":"https://github.com/dsaidgovsg/airflow-pipeline","clone":"git clone https://github.com/dsaidgovsg/airflow-pipeline.git","description":"An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR","language":"Python","stars":176,"topics":["airflow","hadoop","spark","docker"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Airflow Pipeline Docker Image Set-up This repo is a GitHub Actions build matrix set-up to generate Docker images of Airflow, and other major applications as below: - Airflow - Spark - Hadoop integration with Spark - Python - SQL Alchemy Additionally, poetry is used to perform all Python related installations at a predefined global project directory, so that it is easy to add on new packages without conflicting dependency package versions, which raw pip cannot achieve. See for more information. For builds involving Airflow v2 onwards, note that poetry is not officially supported as an installation tool, but it is used anyway to make sure dependencies are compatible and tested to work across multiple builds with different versions. See for a related discussion and how to resolve possible conflicts when installing packages on top of this base image. Entrypoint Also, for convenience, the current version runs both the webserver and scheduler together in the same instance by the default entrypoint, with the webserver being at the background and scheduler at the foreground. All the convenient environment variables only works on the basis that the entrypoint is used without any extra command. If there is a preference to run the various Airflow CLI services separately, you can simply pass the full command into the Docker command, but it will no longer trigger any of the convenient environment variables / functionalities. The above convenience functionalities include: 1. Discovering if","default_branch":null,"files":null,"tree":[],"storefront":"/r/dsaidgovsg","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/dsaidgovsg/airflow-pipeline/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}