{"repo":"dbt-labs/spark-utils","free":true,"listed":false,"github":"https://github.com/dbt-labs/spark-utils","clone":"git clone https://github.com/dbt-labs/spark-utils.git","description":"Utility functions for dbt projects running on Spark","language":"Python","stars":36,"topics":["dbt","spark","macros"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"This dbt package contains macros that: - can be (re)used across dbt projects running on Spark - define Spark-specific implementations of dispatched macros from other packages Installation Instructions Check dbt Hub for the latest installation instructions, or read the docs for more information on installing packages. ---- Compatibility This package provides \"shims\" for: - dbt utils, except for: - dbt utils.get relations by pattern - dbt utils.groupby - dbt utils.recency - dbt utils.any value - dbt utils.listagg - dbt utils.pivot with apostrophe(s) in the values - snowplow (tested on Databricks only) In order to use these \"shims,\" you should set a dispatch config in your root project (on dbt v0.20.0 and newer). For example, with this project setting, dbt will first search for macro implementations inside the spark utils package when resolving macros from the dbt utils namespace: Note to maintainers of other packages The spark-utils package may be able to provide compatibility for your package, especially if your package leverages dbt-utils macros for cross-database compatibility. This package does not need to be specified as a dependency of your package in packages.yml . Instead, you should encourage anyone using your package on Apache Spark / Databricks to: - Install spark utils alongside your package - Add a dispatch config in their root project, like the one above ---- Useful macros: maintenance Caveat: These are not tested in CI, or guaranteed to work on all platforms. Eac","default_branch":null,"files":null,"tree":[],"storefront":"/r/dbt-labs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/dbt-labs/spark-utils/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}