{"repo":"AlvaroCavalcante/airflow-parse-bench","free":true,"listed":false,"github":"https://github.com/AlvaroCavalcante/airflow-parse-bench","clone":"git clone https://github.com/AlvaroCavalcante/airflow-parse-bench.git","description":"Stop creating bad DAGs! Use this tool to measure and compare the parse time of your DAGs, identify bottlenecks, and optimize your Airflow environment for better performance.","language":"Python","stars":23,"topics":["airflow","apache-airflow","dags","data-engineering","python","python3"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Airflow Dag Parse Benchmarking Stop creating bad DAGs! Use this tool to measure and compare the parse time of your DAGs, identify bottlenecks, and optimize your Airflow environment for better performance. Contents - How it works - Installation - Install your Airflow dependencies - Configure your Airflow Variables - Init the Airflow database - Usage - Additional Options - Roadmap - Contribute How It Works Retrieving parse metrics from an Airflow cluster is straightforward, but measuring the effectiveness of code optimizations can be tedious. Each code change requires redeploying the Python file to your cloud provider, waiting for the DAG to be parsed, and then extracting a new report — a slow and time-consuming process. This tool simplifies the process of measuring and comparing DAG parse times. It uses the same parse method as Airflow (from the Airflow repository) to measure the time taken to parse your DAGs locally, storing results for future comparisons. To know more about how the tool works, check out the Medium article. Installation It's recommended to use a virtualenv to avoid library conflicts. Once set up, you can install the package by running the following command: Install your Airflow dependencies The command above installs only the essential library dependencies (Airflow and Airflow providers). You’ll need to manually install any additional libraries that your DAGs depend on. For example, if a DAG uses to interact with AWS, ensure that boto3 is installed in your en","default_branch":null,"files":null,"tree":[],"storefront":"/r/AlvaroCavalcante","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/AlvaroCavalcante/airflow-parse-bench/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}