{"repo":"LewisDavies/upstream-prod","free":true,"listed":false,"github":"https://github.com/LewisDavies/upstream-prod","clone":"git clone https://github.com/LewisDavies/upstream-prod.git","description":"A dbt package for easily using production data in a development environment.","language":"Shell","stars":51,"topics":["dbt","dbt-packages"],"license":"MIT","category":"data-pipelines","readme_excerpt":"What is upstream-prod? upstream-prod is a dbt package for easily using production data in a development environment. It's a hands-off alternative to the defer flag - only without the need to find and download a production manifest - and was inspired by similar work by Monzo. Why do I need it? In a typical project, prod and dev models are materialised in separate environments. Although this ensures end users are unaffected by ongoing development, there's a significant downside: the isolation means that each environment needs a complete, up-to-date copy of every model. This can be challenging for complex projects or long-running models, and out-of-date data can cause frustrating errors. upstream-prod solves this by intelligently redirecting ref s to prod outputs. It is highly adaptable and can be used whether your environments are in separate schemas, databases, or a combination of both. On most warehouses it can even compare dev and prod outputs and use the most recently-updated relation. Known limitations microbatch on Snowflake with dbt Fusion is supported but may be fragile. Fusion's microbatch handling is still a work in progress. It doesn't apply batch filters to refs the way dbt-core does (dbt-labs/dbt-fusion#1608), so upstream-prod reconstructs the filter manually from model.batch and the parent's event time . This works on Snowflake Fusion today, but it leans on Fusion-internal state with no stability guarantees and could break in future Fusion releases. Don't use micr","default_branch":null,"files":null,"tree":[],"storefront":"/r/LewisDavies","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/LewisDavies/upstream-prod/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}