{"repo":"gouline/dbt-metabase","free":true,"listed":false,"github":"https://github.com/gouline/dbt-metabase","clone":"git clone https://github.com/gouline/dbt-metabase.git","description":"dbt + Metabase integration","language":"Python","stars":610,"topics":["analytics","business-intelligence","data","data-modelling","dbt","elt","metabase","pypa","python","vizualisation"],"license":"MIT","category":"data_governance_tool","readme_excerpt":"# dbt-metabase\n\n[![PyPI](https://img.shields.io/pypi/v/dbt-metabase)](https://pypi.org/project/dbt-metabase/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/gouline/dbt-metabase/blob/main/LICENSE)\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)\n\nIntegration between [dbt](https://www.getdbt.com/) and [Metabase](https://www.metabase.com/).\n\nIf dbt is your source of truth for database schemas and you use Metabase as your analytics tool, dbt-metabase can propagate table relationships, model and column descriptions and semantic types (e.g. currency, category, URL) to your Metabase data model, and extract questions and dashboards as exposures in your dbt project.\n\n## Requirements\n\nRequires Python 3.10 or above.\n\nFor development, you will need [uv](https://docs.astral.sh/uv/getting-started/installation/) installed.\n\n## Usage\n\nYou can install dbt-metabase from [PyPI](https://pypi.org/project/dbt-metabase/):\n\n```\npip install dbt-metabase\n```\n\nSections below demonstrate basic usage examples, for all CLI options:\n\n```\ndbt-metabase --help\n```\n\n## Manifest\n\nBefore running dbt-metabase, you need a compiled `manifest.json` file to parse. These are part of the [dbt artifact](https://docs.getdbt.com/reference/artifacts/dbt-artifacts) generated during compilation.\n\nOnce `dbt compile` finishes, `manifest.json` can be found in the `target/` directory of your dbt project.\n\nSee [dbt documentation](https://docs.getdbt.com/docs/running-a-dbt-project/run-your-dbt-projects) for more information.\n\n## Metabase API\n\nAll commands require authentication against the [Metabase API](https://www.metabase.com/docs/latest/api-documentation) using one of these methods:\n\n* API key (`--metabase-api-key`) \n  - Strongly **recommended** for automation, see [documentation](https://www.metabase.com/docs/latest/people-and-groups/api-keys) (Metabase 49 or later).\n* Username and password (`--metabase-username` / `--metabase-password`)\n  - Fallback for older versions of Metabase and smaller instances.\n\n## Exporting Models\n\nLet's start by defining a short sample `schema.yml` as below.\n\n```yaml\nmodels:\n  - name: stg_users\n    description: User records.\n    columns:\n      - name: id\n        description: Primary key.\n        data_tests:\n          - not_null\n          - unique\n\n      - name: email\n        description: User's email address.\n\n      - name: group_id\n        description: Foreign key to user group.\n        data_tests:\n          - not_null\n          - relationships:\n              arguments:\n                to: ref('groups')\n                field: id\n\n  - name: stg_groups\n    description: User groups.\n    columns:\n      - name: id\n        description: Primary key.\n        data_tests:\n          - not_null\n          - unique\n\n      - name: name\n        description: Group name.\n```\n\nThis is already enough to propagate the primary keys, foreign keys and descriptions to Metabase:\n\n```\ndbt-metabase models \\\n    --manifest-path target/manifest.json \\\n    --metabase-url https://metabase.example.com \\\n    --metabase-api-key mb_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX= \\\n    --metabase-database business \\\n    --include-schemas public\n```\n\nOpen Metabase and go to Settings > Admin Settings > Table Metadata, you will notice that `id` column in `stg_users` is now marked as \"Entity Key\" and `group_id` is a \"Foreign Key\" pointing to `id` in `stg_groups`.\n\nTry running `dbt-metabase models --help` to see all the options available for fine tuning.\n\n### Foreign Keys\n\nNative [relationship tests](https://docs.getdbt.com/reference/resource-properties/data-tests#relationships) and [column-level constraints](https://docs.getdbt.com/reference/resource-properties/constraints#defining-constraints) are the recommended ways of defining foreign keys, however you can override them with `fk_target_table` and `fk_target_field` meta fields. If both are set for a column, meta fields take precedence.\n\n```yaml\n- name: country_id\n  description: FK to User's country in the dim_countries table.\n  config:\n    meta:\n      metabase.fk_target_table: analytics_dims.dim_countries\n      metabase.fk_target_field: id\n```\n\nYou can provide `fk_target_table` as `schema_name.table_name` or just `table_name` to use the current schema. If your model has an alias, provide that alias rather than the original name.\n\n### Semantic Types\n\nNow that we have foreign keys configured, let's tell Metabase that `email` column contains email addresses:\n\n```yaml\n- name: email\n  description: User's email address.\n  config:\n    meta:\n      metabase.semantic_type: type/Email\n```\n\nOnce you run `dbt-metabase models` again, you will notice that `email` column is now marked as \"Email\".\n\nBelow are common semantic types (formerly known as _special types_) accepted by Metabase:\n\n* `type/PK`\n* `type/FK`\n* `type/Number`\n* `type/Currency`\n* `type/Category`\n* `type/Title`\n* `type/Description`\n* `type/City`\n* `type/State`\n* `type/ZipCode`\n* `type/Country`\n* `type/Latitude`\n* `type/Longitude`\n* `type/Email`\n* `type/URL`\n* `type/ImageURL`\n* `type/SerializedJSON`\n* `type/CreationTimestamp`\n\nSee [Metabase documentation](https://www.metabase.com/docs/latest/users-guide/field-types.html) for a more complete list.\n\n### Visibility Types\n\nYou can optionally specify visibility for tables and columns, this controls whether they are displayed in Metabase.\n\nHere is how you would hide that email column:\n\n```yaml\n- name: email\n  description: User's email address.\n  config:\n    meta:\n      metabase.semantic_type: type/Email\n      metabase.visibility_type: sensitive\n```\n\nBelow are the visibility types supported for columns:\n\n* `normal` (default) - This field will be displayed normally in tables and charts.\n* `details-only` - This field will only be displayed when viewing the details of a single record.\n* `sensitive` - This field won't be visible or selectable in questions created with the GUI interfaces.\n\nTables support the following:\n\n* No value for visible (default)\n* `hidden`\n* `technical`\n* `cruft`\n\nIf you notice any changes to these, please submit a pull request with an update.\n\n### Other Meta Fields\n\nIn addition to foreign keys, semantic types and visibility types, Metabase also accepts the following meta fields:\n\n```yaml\n- name: model_name\n  config:\n    meta:\n      metabase.display_name: another_model_name\n      metabase.visibility_type: normal\n      metabase.points_of_interest: Relevant records.\n      metabase.caveats: Sensitive information about users.\n  columns:\n    - name: column_name\n      config:\n        meta:\n          metabase.display_name: another_column_name\n          metabase.visibility_type: sensitive\n          metabase.semantic_type: type/Number\n          metabase.has_field_values: list\n          metabase.coercion_strategy: keyword\n          metabase.number_style: decimal\n          metabase.decimals: 3\n```\n\nSee [Metabase documentation](https://www.metabase.com/docs/latest/api) for details and accepted values.\n\n### Synchronization\n\nBy default, dbt-metabase waits for tables and columns to be synchronized between your dbt project and Metabase database, otherwise the export fails when the sync timeout expires. \n\nIf you have known discrepancies between dbt and Metabase and wish to proceed without synchronization, set the sync timeout to zero (e.g. `--sync-timeout 0`). This is discouraged, because you will still encounter errors if you have a table or column in your dbt project that is missing from Metabase and dbt-metabase attempts to export it.\n\n## Exposure Extraction\n\ndbt-metabase allows you to extract questions and dashboards from Metabase as [dbt exposures](https://docs.getdbt.com/docs/building-a-dbt-project/exposures) in your project:\n\n```\ndbt-metabase exposures \\\n    --manifest-path ./target/manifest.json \\\n    --metabase-url https://metabase.example.com \\\n    --metabase-api-key mb_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX= \\\n    --output-path models/ \\\n    --exclude-collections \"temp*\"\n```\n\nOnce the execution completes, check your output path for exposures files containing descriptions, creator details and links for Metabase questions and dashboards:\n\n```yaml\nexposures:\n  - name: number_of_orders_over_time\n    description: '\n      ### Visualization: Line\n\n      A line chart depicting how order volume changes over time\n\n      #### Metadata\n\n      Metabase Id: __8__\n\n      Created On: __2021-07-21T08:01:38.016244Z__'\n    type: analysis\n    url: https://metabase.example.com/card/8\n    maturity: medium\n    owner:\n      name: Indiana Jones\n      email: indiana@example.com\n    depends_on:\n      - ref('orders')\n```\n\nNative query questions will have SQL code blocks inside the descriptions, formatted to look nice in [dbt docs](https://docs.getdbt.com/docs/collaborate/documentation). These YAML files can be committed to source control to understand how exposures change over time.\n\nTry running `dbt-metabase exposures --help` to see all the options available for fine tuning.\n\n## Configuration\n\nThere are 3 levels of configuration in decreasing order of precedence:\n\n* CLI arguments, e.g. `--manifest-path target/manifest.json`\n* Environment variables, e.g. `MANIFEST_PATH=target/manifest.json`\n* Configuration file, e.g. `manifest_path: target/manifest.json`\n\nTry running `--help` for any command to see the full list of CLI arguments and environment variables.\n\nA configuration file can be created in `~/.dbt-metabase/config.yml` for dbt-metabase to pick it up automatically or anywhere else by specifying `dbt-metabase --config-path path/to/config.yml` (must come **before** the command). Here is an example YAML file:\n\n```yaml\nconfig:\n    manifest_path: target/manifest.json\n    metabase_url: https://metabase.example.com\n    metabase_api_key: mb_XXXXXX","default_branch":"main","files":108,"tree":[".github/FUNDING.yml",".github/workflows/codeql-analysis.yml",".github/workflows/lock_threads.yml",".github/workflows/main.yml",".github/workflows/pull_request.yml",".gitignore","CONTRIBUTING.md","LICENSE","Makefile","README.md","dbtmetabase/__init__.py","dbtmetabase/__main__.py","dbtmetabase/_exposures.py","dbtmetabase/_models.py","dbtmetabase/core.py","dbtmetabase/errors.py","dbtmetabase/format.py","dbtmetabase/manifest.py","dbtmetabase/metabase.py","pyproject.toml","sandbox/.env","sandbox/.gitignore","sandbox/Dockerfile","sandbox/README.md","sandbox/dbt_project.yml","sandbox/docker-compose.yml","sandbox/entrypoint.py","sandbox/metabase.db/metabase.db.mv.db","sandbox/models/.gitignore","sandbox/models/customers.sql","sandbox/models/orders.sql","sandbox/models/payments.sql","sandbox/models/schema.yml","sandbox/models/staging/schema.yml","sandbox/models/staging/stg_customers.sql","sandbox/models/staging/stg_orders.sql","sandbox/models/staging/stg_payments.sql","sandbox/postgres-initdb/init.sql","sandbox/profiles.yml","sandbox/seeds/raw_customers.csv","sandbox/seeds/raw_orders.csv","sandbox/seeds/raw_payments.csv","tests/__init__.py","tests/_mocks.py","tests/conftest.py","tests/fixtures/manifest-cross-schema-fk.json","tests/fixtures/manifest-v11-disabled.json","tests/fixtures/manifest-v12.json","tests/fixtures/manifest-v2.json","tests/fixtures/mbql4/api/card/27.json","tests/fixtures/mbql4/api/card/28.json","tests/fixtures/mbql4/api/card/29.json","tests/fixtures/mbql4/api/card/30.json","tests/fixtures/mbql4/api/card/31.json","tests/fixtures/mbql4/api/card/32.json","tests/fixtures/mbql4/api/card/33.json","tests/fixtures/mbql4/api/collection.json","tests/fixtures/mbql4/api/collection/3/items.json","tests/fixtures/mbql4/api/collection/root/items.json","tests/fixtures/mbql4/api/dashboard/2.json","tests/fixtures/mbql4/api/database.json","tests/fixtures/mbql4/api/database/2/metadata.json","tests/fixtures/mbql4/api/table.json","tests/fixtures/mbql4/api/user/1.json","tests/fixtures/mbql4/exposure/collection/our_analytics.yml","tests/fixtures/mbql4/exposure/collection/u043a_u043e_u043b_u043b_u0435_u043a_u0446_u0438_u044f.yml","tests/fixtures/mbql4/exposure/default/exposures.yml","tests/fixtures/mbql4/exposure/type/card/27.yml","tests/fixtures/mbql4/exposure/type/card/28.yml","tests/fixtures/mbql4/exposure/type/card/29.yml","tests/fixtures/mbql4/exposure/type/card/30.yml","tests/fixtures/mbql4/exposure/type/card/31.yml","tests/fixtures/mbql4/exposure/type/card/32.yml","tests/fixtures/mbql4/exposure/type/card/33.yml","tests/fixtures/mbql4/exposure/type/dashboard/2.yml","tests/fixtures/mbql5/api/card/27.json","tests/fixtures/mbql5/api/card/28.json","tests/fixtures/mbql5/api/card/29.json","tests/fixtures/mbql5/api/card/30.json","tests/fixtures/mbql5/api/card/31.json","tests/fixtures/mbql5/api/card/32.json","tests/fixtures/mbql5/api/card/33.json","tests/fixtures/mbql5/api/collection.json","tests/fixtures/mbql5/api/collection/3/items.json","tests/fixtures/mbql5/api/collection/root/items.json","tests/fixtures/mbql5/api/dashboard/2.json","tests/fixtures/mbql5/api/database.json","tests/fixtures/mbql5/api/database/2/metadata.json","tests/fixtures/mbql5/api/table.json","tests/fixtures/mbql5/api/user/1.json","tests/fixtures/mbql5/exposure/collection/our_analytics.yml","tests/fixtures/mbql5/exposure/collection/u043a_u043e_u043b_u043b_u0435_u043a_u0446_u0438_u044f.yml","tests/fixtures/mbql5/exposure/default/exposures.yml","tests/fixtures/mbql5/exposure/type/card/27.yml","tests/fixtures/mbql5/exposure/type/card/28.yml","tests/fixtures/mbql5/exposure/type/card/29.yml","tests/fixtures/mbql5/exposure/type/card/30.yml","tests/fixtures/mbql5/exposure/type/card/31.yml","tests/fixtures/mbql5/exposure/type/card/32.yml","tests/fixtures/mbql5/exposure/type/card/33.yml","tests/fixtures/mbql5/exposure/type/dashboard/2.yml","tests/fixtures/test_dump_yaml.yml","tests/test_exposures.py","tests/test_format.py","tests/test_manifest.py","tests/test_metabase.py","tests/test_models.py","uv.lock"],"storefront":"/r/gouline","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/gouline/dbt-metabase/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}