{"repo":"gmyrianthous/dbt-airflow","free":true,"listed":false,"github":"https://github.com/gmyrianthous/dbt-airflow","clone":"git clone https://github.com/gmyrianthous/dbt-airflow.git","description":"A Python package that creates fine-grained dbt tasks on Apache Airflow","language":"Python","stars":85,"topics":["airflow","analytics-engineering","data-engineering","dbt","python"],"license":null,"category":"data-pipelines","readme_excerpt":"dbt-airflow A Python package that helps Data and Analytics engineers render dbt projects in Apache Airflow DAGs such that models, seeds, snapshots and tests are represented by individual Airflow Task. dbt is a command-line tool that enables data teams build, maintain and test data models in a scalable fashion. The biggest challenge though is how to embed dbt in modern data workflows and infrastructure. dbt CLI is indeed a powerful tool, but if used as is, it will create silos in the way an organisation manages its data. Every contributor is able to run dbt commands from their local machine (or even a host machine), but how do you know if a model run by another contributor has failed, or succeeded? How can you enable shared visibility over data models, within the team? One way to host dbt projects and orchestrate dbt tasks is via Apache Airflow. In its simplest form, an Airflow DAG that will build and test data models will consist of two tasks, one that executes dbt run command followed by an Airflow task that executes dbt test . But what happens when model builds or tests fail? Should we re-run the whole dbt project (that could involve hundreds of different models and/or tests) just to run a single model we've just fixed? This doesn't seem to be a good practice since re-running the whole project will be time-consuming and expensive. A potential solution to this problem is to create individual Airflow tasks for every model, seed, snapshot and test within the dbt project. If we","default_branch":null,"files":null,"tree":[],"storefront":"/r/gmyrianthous","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/gmyrianthous/dbt-airflow/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}