{"repo":"mwojtyczka/databricks-dbt-factory","free":true,"listed":false,"github":"https://github.com/mwojtyczka/databricks-dbt-factory","clone":"git clone https://github.com/mwojtyczka/databricks-dbt-factory.git","description":"Lightweight Python library and CLI that generates Databricks Workflows definition from your dbt project — each dbt model, test, seed, and snapshot runs as a separate task.","language":"Python","stars":36,"topics":["data-engineering","data-pipelines","databricks","databricks-asset-bundles","databricks-workflows","dbt","dbt-core","elt","python","workflow-orchestration"],"license":"MIT","category":"data-pipelines","readme_excerpt":"Databricks dbt factory === Databricks dbt-factory is a lightweight library and CLI that turns a dbt project into a granular Databricks Workflow — one task per dbt object (models, tests, seeds, and snapshots) instead of a single opaque dbt task. It reads your dbt manifest and generates a new job specification — a Databricks Asset Bundle (DAB) or plain job YAML — or updates an existing one in place. ----- Table of Contents - Motivation - How it works - Installation - Usage - Handling dbt tests - Task types - End-to-end example - Contribution - License Motivation By default, running a dbt project in Databricks Workflows treats an entire dbt project as a single execution unit — a black box. Databricks dbt-factory changes that by updating Databricks Workflow specs to run dbt objects (models, tests, seeds, snapshots) as individual tasks. The diagram below shows an example dbt project structure to illustrate the idea — the actual task graph is derived from your dbt manifest. Benefits ✅ Faster execution — run dbt objects in parallel across Databricks tasks instead of one sequential run. ✅ Visibility & Simplified troubleshooting — Quickly pinpoint and fix issues at the model level. ✅ Enhanced logging & notifications — Gain detailed logs and precise error alerts for faster debugging. ✅ Improved retriability — Retry only the failed model tasks without rerunning the full project. ✅ Seamless testing — Automatically run dbt data tests on tables right after each model finishes, enabling fas","default_branch":null,"files":null,"tree":[],"storefront":"/r/mwojtyczka","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mwojtyczka/databricks-dbt-factory/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}