Create Databricks data source

1. Navigate to Settings > Data Ingestion > Data Sources.
2. Click New data source.
   • 
3. Select your system and the newly created connection.
4. Name your source (and add a description) for easier identification in Dawiso.

Scope the scan

Define which parts of the metastore to catalog. All fields are optional — leave them empty to scan everything the connection’s principal can see:

1. Catalogs — include list: Catalogs to scan, e.g. ["prod", "finance"]. Empty = all visible catalogs.
2. Catalogs — exclude list: Catalogs to skip.
3. Qualified schemas — include list: Per-catalog schema scoping with catalog.schema entries; * wildcards are supported per segment, e.g. ["prod.gold", "finance.*"].
4. Qualified schemas — exclude list: Same syntax as including schemas, subtracted from scope.

Lineage settings

1. Lineage Days: Define how far back Dawiso reads system.access lineage events (default 30). Longer windows catch rarely-run flows; shorter windows scan faster.
2. Lineage Workspace Scope: Restrict lineage events to the connected workspace or ingest metastore-wide events.

Optional scans

Toggle the object families you want catalogued (everything is on by default except storage credentials):

• Constraints (primary/foreign keys — needs USE CATALOG on scanned catalogs)
• Dashboards (AI/BI dashboards + their datasets — needs CAN VIEW)
• Genie Spaces (+ sample queries — needs CAN VIEW)
• Tags, Column Masks, Row Filters, Data Classification
• ML Models (needs per-model EXECUTE or workspace admin)
• External Locations
• Storage Credentials (off by default — needs metastore admin)

Advanced: Statement timeout and HTTP timeout (seconds) can be raised for very large metastores.

Destination configuration

In the Destination configuration step:

1. Select into what space you want to store your ingested metadata. You can select only the spaces you have access to.
2. Select the workflow for ingested objects management. For more information on workflows, see the article on Workflow types.
3. [Optional steps]
   • Schedule: To customize the regular automated ingestion, check the box next to Schedule and adjust the frequency.
   • Optional Settings: Specify options supported by the provider in JSON format. Option will be made available based on feedback. For more information, see Optional Settings.
   • Additional Settings: Add a JSON formatted list of additional settings that can be used to troubleshoot your custom scenarios. We recommend leaving this field empty unless our Dawiso support team suggests otherwise.
   • Enable AI-Generated Descriptions: Enable AI-generated descriptions for your ingested metadata. Descriptions will be generated according to prompts that are defined in packages. Each attribute type can have its own configurable prompt and any AI-generated content will be clearly marked with a banner the text is manually reviewed and saved.
     • Descriptions will be generated on the following levels: database, schema, view, and table.
4. Save your data source.

Optional settings

In the data source configuration step, you can specify a query transformation in the Optional Settings field. This allows for dynamic adjustments to queries based on predefined values.

Query transformations can access the following data source fields:

• optionalSettings.dictionary: Users can define key-value pairs to be referenced in transformations.

  "dictionary":{ "schemas": "dbo, system, api", "databases": "Alpha, Bravo" }

• datasource: Users can reference values directly from data source properties, which will vary by provider.

  datasource.databases

• optionalSettings: Users can reference values from optional settings in the data source configuration.

    ```json

  { “databases”: “Alpha, Bravo” }

Run ingestion

Once you create your data source, you will be redirected to a list of your data sources.

1. Click the three dots next to your newly created source, and select Run ingestion.
2. Confirm to ingest your data.

You can find all new, running, or completed ingestions by navigating to Settings > Data Ingestion > Ingestions.