{"repo":"arnoN7/dbt-incremental-stream","free":true,"listed":false,"github":"https://github.com/arnoN7/dbt-incremental-stream","clone":"git clone https://github.com/arnoN7/dbt-incremental-stream.git","description":"DBT Package reproducing dbt incremental materialization leveraging on Snowflake streams","language":"Python","stars":38,"topics":["dbt","dbt-packages","snowflake","stream","streaming"],"license":"MIT","category":"data-pipelines","readme_excerpt":"dbt-incremental-stream This dbt package is for Snowflake ❄️ only. It is reproducing dbt incremental materialization leveraging on Snowflake streams to : Improve model performance 💨 . Stream accelerate source table scan focusing new data only Optimise FinOps 💰 . Snowflake warehouse will be started only if stream is not empty. Simplify code. is incremental() is generally not required as stream natively collect \"new\" rows Installation Instruction New to dbt packages? Read more about them here. 1. Include this package in your packages.yml . An example can be found here in the working example project 2. Run dbt deps Usage Use incremental stream materialisation like dbt incremental model : Replace ref by stream ref macro to add a stream on a dbt model Replace source by stream source to add a stream on a source incr stream.get stream metadata columns() must be included to retreive METADATA columns of each STREAMS . Hence the materialization can deal accordingly with the changes on inputs tables ( INSERT , UPDATE , DELETE ) but Metadata columns ( METADATA$ACTION , METADATA$ISUPDATE , METADATA$ROW ID ) will not be included in the model Like in dbt incremental model unique key optional parameter is supported. DBT will perform a MERGE with a unique key an INSERT instead. Like in dbt incremental model --full-refresh rebuild the target model based on the source tables An optional dbt Project Variable can be added to create streams at a specific timestamp using Snowflake Time Travel with","default_branch":null,"files":null,"tree":[],"storefront":"/r/arnoN7","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/arnoN7/dbt-incremental-stream/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}