{"repo":"datopian/aircan","free":true,"listed":false,"github":"https://github.com/datopian/aircan","clone":"git clone https://github.com/datopian/aircan.git","description":"💨🥫 A Data Factory system for running data processing pipelines built on AirFlow and tailored to CKAN. Includes evolution of DataPusher and Xloader for loading data to DataStore.","language":"Python","stars":34,"topics":["ckan-datastore","ckan","airflow","csv","google-cloud-composer","dag","airflow-ui"],"license":null,"category":"data-pipelines","readme_excerpt":"Aircan Aircan is a collection of Apache Airflow DAGs for building and operating data pipelines. It provides reusable components for common data engineering tasks — including cloud storage integration, data warehouse loading, schema management, validation, and notifications — and can be extended to support any data pipeline use case. Repository Structure Triggers Aircan DAGs can be triggered in several ways depending on your use case: - CKAN integration — Install the ckanext-aircan extension, an alternative to DataPusher and XLoader, which automatically triggers the DAG whenever a new CSV resource is added or updated in CKAN. - Scheduled runs — Configure a cron-based schedule directly on the DAG to ingest data at regular intervals (e.g. nightly, hourly). - Manual triggers — Run a DAG on demand via the Airflow UI, passing any required parameters at runtime. - REST API — Trigger DAGs programmatically using the Airflow REST API, suitable for integration with external systems or CI/CD pipelines. Quick Start (Docker Compose) 1. Create a .env file Copy the template below and fill in your values. The Docker Compose file reads this file automatically. Note: .env is git-ignored. Never commit credentials. 3. Start the containers Airflow UI will be available at http://localhost:8080 (default credentials: airflow / airflow ). 4. Register Airflow Connections Each pipeline run is namespaced by a site id (set in the DAG trigger params). Airflow connection IDs follow the pattern {site id} {ty","default_branch":null,"files":null,"tree":[],"storefront":"/r/datopian","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/datopian/aircan/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}