1. Navigate to Settings  > Data Ingestion > Data Sources.
  2. Click New data source.
    • Create data source.png
  3. Select your system and the newly created connection.
  4. Name your source (and add a description) for easier identification in Dawiso.

Select account and project

In this step, define what account to extract metadata from and which projects and environments should be scanned:

  1. Account ID: Your dbt Cloud Account ID, obtainable from the dbt Cloud URL. It is the first number within it.

  2. Projects: Select all projects to ingest to Dawiso by listing them using a regular expression, e.g.:

    Project1|Project2
  3. Environments: Select environments for each project by listing them using a regular expression, e.g.:

    Prod|Test|Dev
  4. Production Only: Enabled to include only environments marked with the PROD deployment type. All others are filtered out.

  5. Jobs: Filter jobs for selected projects and environments using a regular expression.

Editing the list of projects

If you edit the list of projects, environments or jobs for an existing and already ingested data source, note the following:

  • Newly added projects will be ingested during the next ingestion run.
  • Removed projects will be treated as deleted. Dawiso will automatically remove all associated metadata during the next run. This includes any user-added content such as descriptions, ownership, or relations.

Destination configuration

In the Destination configuration step:

  1. Select into what space you want to store your ingested metadata. Keep in mind you can select only the spaces you have access to.

  2. Select the workflow for ingested objects management. For more information on workflows, see the article on Workflow types.

  3. [Optional steps]

    ConfigurationDescription
    ScheduleTo customize the regular automated ingestion, check the box next to Schedule and adjust the frequency.
    Optional SettingsSpecify options supported by the provider in JSON format. Option will be made available based on feedback. For more information, see Optional Settings.
    Additional SettingsAdd a JSON formatted list of additional settings that can be used to troubleshoot your custom scenarios. We recommend leaving this field empty unless our Dawiso support team suggests otherwise.
    Enable AI-Generated DescriptionsEnable AI-generated descriptions for your ingested metadata. Descriptions will be generated according to prompts that are defined in packages. Each attribute type can have its own configurable prompt and any AI-generated content will be clearly marked with a banner the text is manually reviewed and saved. Descriptions will be generated on the following levels: database, schema, view, and table. Enable AI-Gen Descriptions.png
  4. Save your data source.

Warning

If you delete a data source and then create a new one to the same space, the original ingested data will remain and mix with the new data. To prevent duplicates, always delete the metadata application from the space (with the old source) before creating and ingesting a new one.

Run ingestion

Once you create your data source, you will be redirected to a list of your data sources.

  1. Click the three dots next to your newly created source, and select  Run ingestion.
  2. Confirm to ingest your data.

You can find all new, running, or completed ingestions by navigating to Settings  > Data Ingestion > Ingestions. For more information, see the article on Ingestions.

Tip

If you are ingesting data on-premises, refer to these articles instead Dawiso Integration Runtime (DIR).

Warning

Once data is ingested to metadata applications, manual changes to the object hierarchy are NOT allowed. Changes such as adding, deleting, or moving objects under different parents may cause data inconsistencies and break data scans.