Before ingesting Databricks metadata into Dawiso, prepare authentication using one of the following methods:

  • OAuth service principal (Client ID + Client Secret) — recommended
  • Personal Access Token (PAT)
  • Managed Identity (Azure only)

Dawiso gathers Databricks metadata through Unity Catalog system tables (executed on a SQL warehouse) and the Unity Catalog REST API.

Connection prerequisites

  • Unity Catalog is enabled for the workspace
  • A SQL warehouse the scanning principal can use (CAN USE)
  • Databricks system tables are enabled (see below)
  • On AWS and GCP you will also need the numeric Workspace ID (o= value in your Databricks URL)

Get your Databricks workspace URL

Find your workspace URL in your browser’s address bar:

  • https://adb-000XXX000XXX.azuredatabricks.net for Databricks on Azure
  • https://dbc-000XXX000XXX.cloud.databricks.com for Databricks on AWS
  • https://000XXX000XXX.gcp.databricks.com for Databricks on GCP

Create the service principal (OAuth)

Log in to the Databricks Account Console for your cloud:

  • AWS: https://accounts.cloud.databricks.com
  • Azure: https://accounts.azuredatabricks.net
  • GCP: https://accounts.gcp.databricks.com
  1. In the left menu, select User management and go to the Service principals tab.
  2. Click Add service principal, enter a name (e.g. dawiso-integration), and click Add.
  3. Open the service principal and go to the Secrets tab.
  4. Click Generate secret, optionally set an expiry period, and click Generate.
  5. Copy the Client ID (shown as Application ID) and the Client Secret.
    Warning

    The Client Secret is shown only once. Store both values securely before closing the dialog.

  6. Go to Workspaces, select your target workspace, click Permissions > Add permissions, and assign the service principal the User role or higher.

Generate a personal access token (alternative)

  1. In your Databricks account, click your username in the top bar and select Settings.
  2. Click Developer. Next to Access tokens, click Manage.
  3. Click Generate new token, optionally set a name and expiry, and click Generate.
  4. Copy the displayed token and click Done.

Grant privileges

The scanner is read-only and works in privilege tiers — grant the tier you are comfortable with; anything not granted simply isn’t catalogued.

-- metadata visibility (catalog skeleton, filtered to what the principal may see)
GRANT BROWSE ON METASTORE TO `<service-principal-application-id>`;

-- system tables (metadata and telemetry only — none of your table data)
GRANT USE CATALOG ON CATALOG system                  TO `<sp>`;
GRANT SELECT ON SCHEMA system.information_schema     TO `<sp>`;
GRANT SELECT ON SCHEMA system.access                 TO `<sp>`;   -- lineage
GRANT SELECT ON SCHEMA system.lakeflow               TO `<sp>`;   -- jobs, pipelines
GRANT SELECT ON SCHEMA system.compute                TO `<sp>`;   -- SQL warehouses
GRANT SELECT ON SCHEMA system.data_classification    TO `<sp>`;   -- PII classification

-- SQL warehouse
-- grant CAN USE on the warehouse the scanner should run statements on

Each optional system.* grant maps to one feature. Omit a feature to skip it. System-schema grants must be issued by an account admin, and system schemas must be enabled once per metastore. For more information, see the official Databricks documentation.

Scan SQL / YAML definitions and metric-view lineage

BROWSE alone exposes the catalog skeleton including metric-view columns. Definitions and metric-view base lineage need SELECT on the in-scope objects:

GRANT USE CATALOG ON CATALOG <catalog>               TO `<sp>`;
GRANT USE SCHEMA  ON CATALOG <catalog>               TO `<sp>`;
GRANT SELECT      ON CATALOG <catalog>               TO `<sp>`;

If metastore-wide BROWSE feels broad, grant BROWSE per catalog instead and set the data source’s Catalogs — include list to match.

Workspace resources (optional)

Dashboards and Genie spaces are read through the workspace API — the service principal needs CAN VIEW on the dashboards / Genie spaces you want catalogued (or workspace admin). ML models need per-model EXECUTE or workspace admin. These scans are toggles on the data source and skip silently when not granted.

Data lineage requirements

Lineage is read from system.access.table_lineage and system.access.column_lineage (see the grants above). Lineage rows are activity-based: they record executed reads/writes, and the scanner ingests the window configured by Lineage Days on the data source (default 30). Databricks lineage requirements and limitations apply.