{"repo":"himewel/airflow_celery_workers","free":true,"listed":false,"github":"https://github.com/himewel/airflow_celery_workers","clone":"git clone https://github.com/himewel/airflow_celery_workers.git","description":"Run Apache Airflow with CeleryExecutor using Docker Compose. Includes scalable workers, Redis broker, PostgreSQL metadata DB, Flower UI, and scripts for GCP VM deployment with Nginx reverse proxy","language":"Shell","stars":15,"topics":["apache-airflow","celery","docker","data-engineering","distributed-systems","docker-compose","etl","flower","gcp","nginx"],"license":null,"category":"deployment-docker-iac","readme_excerpt":"Airflow Celery Workers Docker Compose setup for Apache Airflow 2.0 with the Celery executor and horizontally scalable workers. Overview This repository runs Airflow with: - PostgreSQL — metadata database - Redis — Celery message broker between the scheduler and workers - Scheduler — orchestrates DAG runs - Webserver — Airflow UI (port 8080) - Flower — Celery monitoring UI (port 5555) - Worker(s) — execute tasks; scale with --scale worker=N Stack versions Component Version ----------- --------- Apache Airflow 2.0.2 PostgreSQL 12 (Alpine) Redis 6.0.9 (Alpine) Prerequisites - Docker - Docker Compose (v1 docker-compose or Compose V2 docker compose ) Quick start (local) Build and start all services with four Celery workers: Start with a single worker (the default): Access the UIs Service URL Default credentials --------- ----- --------------------- Airflow Webserver http://localhost:8080 admin / admin Celery Flower http://localhost:5555/flower/ admin / admin Credentials are defined in the x-default-user section of docker-compose.yaml . Change them before deploying to any shared or production environment. Stop services To also remove database volumes: Project structure GCP deployment On a Google Cloud VM, install host dependencies, start the stack, and configure Nginx as a reverse proxy so the UIs are reachable on port 80. Before running proxy config.sh , edit config/proxy and replace the placeholder internal IPs with the private IPs of the VM(s) running the webserver and Flower co","default_branch":null,"files":null,"tree":[],"storefront":"/r/himewel","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/himewel/airflow_celery_workers/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}