Ingestion with Azure Managed Identity

When you run Dawiso Integration Runtime (DIR) on an Azure host (a virtual machine or AKS pod), you can authenticate to Databricks with the host’s Azure managed identity instead of a stored client secret or personal access token. The token is requested at runtime from Azure and never written to disk or into the configuration file.

Prerequisites

• DIR runs on an Azure VM or AKS pod that has a managed identity assigned.
• Unity Catalog is enabled for the workspace.
• You know the managed identity’s client (application) ID (for a user-assigned identity).

How it works

1. DIR asks Azure for an access token for the Azure Databricks resource using the host’s managed identity.
2. DIR then sends that token to your Databricks workspace as the bearer token.
3. Databricks accepts the token once the identity is registered as a service principal and granted the required privileges.

• User-assigned identity: Set clientId to the identity’s client (application) ID.
• System-assigned identity: Leave clientId empty.

1. Register the managed identity in Databricks

The managed identity must exist in Databricks as a service principal, identified by its Azure Application (client) ID.

1. Open the Databricks Account Console.
2. In the left menu, select User management.
3. Open the Service principals tab and click Add service principal.
4. Choose the Microsoft Entra ID managed type, and paste the managed identity’s client ID as the Application ID.
5. Go to Workspaces, select your target workspace, click Permissions > Add permissions, find the service principal, and assign it the User role or higher.

2. Grant required privileges

Grant the service principal access to the catalogs and schemas you want to ingest. Replace <client-id> with the managed identity’s application ID.

GRANT USE CATALOG ON CATALOG <catalog_name> TO `<client-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<client-id>`;
GRANT SELECT ON SCHEMA <catalog_name>.<schema_name> TO `<client-id>`;
GRANT EXECUTE ON SCHEMA <catalog_name>.<schema_name> TO `<client-id>`;

To ingest data lineage, grant the service principal access to the system.access schema, as described in Prepare for Databricks integration.

3. Configure DIR

Use the core_databricks_v2 provider and set authMode to managed_identity in the connection block. No secret is required.

{
  "general": {
    "workingFolder": "working_folder",
    "extractOnly": false
  },
  "ingestionCloud": {
    "urlAddress": "https://<your-ingestion-cloud-url>/",
    "apiToken": "<api-token>",
    "timeout": "00:20:00"
  },
  "dataSource": {
    "providerKey": "core_databricks_v2",
    "uuid": "<datasource UUID>",
    "format": "full",
    "connection": {
      "hostname": "adb-000XXX000XXX.azuredatabricks.net",
      "warehouseId": "<warehouseId>",
      "authMode": "managed_identity",
      "clientId": "<managed-identity-client-id>"
    },
    "settings": {
      "warehouseId": "<warehouseId>",
      "catalogNames": "<catalogName>",
      "loadTableLineageViaStatements": true,
      "loadColumnLineageViaStatements": true,
      "loadVolumeLineage": true
    }
  }
}

Field	Notes
providerKey	Must be core_databricks_v2. The older core_databricks provider does not support managed identity.
connection.hostname	Workspace host without scheme, e.g. adb-000XXX000XXX.azuredatabricks.net.
connection.authMode	managed_identity. Other modes: oauth (Client ID + Secret) and pat (Personal Access Token).
connection.clientId	User-assigned identity’s client ID. Leave empty for a system-assigned identity.
connection.warehouseId	SQL warehouse used for lineage queries.

4. Run the ingestion

Run DIR with your configuration file:

DawisoIntegrationRuntime --config <path-to-config>.json

The run finishes with Processing finished successfully once the identity is authenticated and authorized. Empty result warnings (for example ... has no result for lineage or classifications) are normal when a catalog has no such objects.

Troubleshooting

Test ingestion returns 403

If the test ingestion returns 403 on unity-catalog/metastore_summary, the identity is authenticated but lacks Unity Catalog access.

Ensure that:

• The service principal is assigned to the workspace,
• A metastore is attached, and
• The grants above are in place