{"repo":"mozilla/telemetry-airflow","free":true,"listed":false,"github":"https://github.com/mozilla/telemetry-airflow","clone":"git clone https://github.com/mozilla/telemetry-airflow.git","description":"Airflow configuration for Telemetry","language":"Python","stars":205,"topics":["airflow","mozilla-telemetry"],"license":"MPL-2.0","category":"data-pipelines","readme_excerpt":"Telemetry-Airflow Apache Airflow is a platform to programmatically author, schedule and monitor workflows. - Telemetry-Airflow - Contributing - Writing DAGs - Prerequisites - Installing dependencies locally - Updating Python dependencies - Build Container - macOS - Testing - Local Deployment - Adding dummy credentials - Usage - Testing GKE Jobs (including BigQuery-etl changes) - Testing Dataproc Jobs - Debugging - Production Setup - Deployments This repository codifies the Airflow cluster that is deployed at workflow.telemetry.mozilla.org (behind SSO) and commonly referred to as \"WTMO\" or simply \"Airflow\". Some links relevant to users and developers of WTMO: - The dags directory in this repository contains some custom DAG definitions - Many of the DAGs registered with WTMO don't live in this repository, but are instead generated from ETL task definitions in bigquery-etl - The Data SRE team maintains a WTMO Developer Guide (behind SSO) Contributing Forking workflow is required ONLY for contributors without write access . This repo enforces Conventional Commit style via the Github action Semantic PRs. Writing DAGs See the Airflow's Best Practices guide to help you write DAGs. ⚠ Warning: Do not import resources from the dags directory in DAGs definition files ⚠ As an example, if you have dags/dag a.py and dags/dag b.py and want to use a helper function in both DAG definition files, define the helper function in the utils directory such as: utils/helper.py dags/dag a.py dags/dag ","default_branch":null,"files":null,"tree":[],"storefront":"/r/mozilla","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mozilla/telemetry-airflow/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}