{"repo":"mozilla/python_mozetl","free":true,"listed":false,"github":"https://github.com/mozilla/python_mozetl","clone":"git clone https://github.com/mozilla/python_mozetl.git","description":"ETL jobs for Firefox Telemetry","language":"Python","stars":29,"topics":["etl","pyspark","python","mozilla-telemetry"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Firefox Telemetry Python ETL This repository is a collection of ETL jobs for Firefox Telemetry. Benefits Jobs committed to python mozetl can be scheduled via airflow or ATMO . We provide a testing suite and code review , which makes your job more maintainable. Centralizing our jobs in one repository allows for code reuse and easier collaboration . There are a host of benefits to moving your analysis out of a Jupyter notebook and into a python package. For more on this see the writeup at cookiecutter-python-etl. Tests Dependencies First install the necessary runtime dependencies -- snappy and the java runtime environment. These are used for the pyspark package. In ubuntu: Calling the test runner Run tests by calling tox in the root directory. Arguments to pytest can be passed through tox using -- . Tests are configured in tox.ini Manual Execution ATMO The first method of manual execution is the mozetl-submit.sh script located in bin . This script is used with the EMRSparkOperator in telemetry-airflow to schedule execution of mozetl jobs. It may be used with ATMO to manually test jobs. In an SSH session with an ATMO cluster, grab a copy of the script: Push your code to your own fork, where the job has been added to mozetl.cli . Then run it. See comments in bin/mozetl-submit.sh for more details. Databricks Jobs may also be executed on Databricks. They are scheduled via the MozDatabricksSubmitRunOperator in telemetry-airflow . This script runs on your local machine and submits th","default_branch":null,"files":null,"tree":[],"storefront":"/r/mozilla","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mozilla/python_mozetl/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}