Prepare for Google BigQuery integration

Before ingesting Google BigQuery metadata into Dawiso, prepare your account for authentication by configuring a Google BigQuery service account and granting it the necessary permissions.

Dawiso queries INFORMATION_SCHEMA views to get metadata for your assets.

Connection configuration

In this guide, you will:

• Create a Service Role and configure its permissions
• Create a dedicated Service Account and add it to the newly created role
• Generate a Service Account Key in JSON
• [Optional] Enable the Data Lineage API and grant the Data Lineage Viewer role

This guide assumes you are using the Google Cloud console, but all steps can also be carried out in the Google Cloud CLI.

Create service role

First, create a new Identity and Access Management (IAM) role.

1. In your Google Cloud console, open the left-side menu, hover over IAM and admin, and click Roles. Make sure you’ve selected the correct project.
2. In the top panel, click + CREATE ROLE. On the next page, fill in the following fields:
   1. Title: Name the role, e.g., Dawiso Ingestion role.
   2. [Optional] Description: Add a description to help identify the role and its purpose.
   3. ID: Unique ID of the role.
   4. [Optional] Role launch stage: Select a role launch stage. Available options are: Alpha, Beta, or General Availability.
3. On the same page, click + ADD PERMISSIONS, select all the following permissions, and click Add:

   •   bigquery.datasets.get
       bigquery.datasets.getIamPolicy
       bigquery.jobs.create
       bigquery.readsessions.create
       bigquery.readsessions.getData
       bigquery.readsessions.update
       bigquery.routines.get
       bigquery.routines.list
       bigquery.tables.get
       bigquery.tables.getIamPolicy
       bigquery.tables.list
       resourcemanager.projects.get
       resourcemanager.projects.getIamPolicy

4. Finally, finish the role setup by clicking Create.

Create service account

Next, create a Service Account to grant Dawiso access to your project. Please follow this guide.

1. In your Google Cloud console, open the left-side menu, hover over IAM and admin, and click Service Accounts. Make sure you’ve selected the correct project.
2. In the top panel, click + CREATE SERVICE ACCOUNT. On the next page, fill in the following fields:
   1. Service account name: Name the service account, e.g., Dawiso service account.
   2. Service account ID: Unique ID of the service account. The ID will be auto-generated, however, you can change it if needed.
   3. [Optional] Service account description: Add a description to help identify the role and its purpose.
3. Click CREATE AND CONTINUE.
4. In the Grant this service account access to project section, select the newly created service role (in this case, Dawiso Ingestion role).
5. Click Continue.
6. Finish the service account setup by clicking DONE.

Generate a key for the service account

Finally, you will need to generate a JSON Service Account Key that allows access to your cloud resources to create a new Connection in Dawiso.

1. In your Google Cloud console, open the left-side menu, hover over IAM and admin, and click Service Accounts. Make sure you’ve selected the correct project.
2. Click the three dots icon next to the service account created for Dawiso and select Manage keys.
3. Click ADD KEY above the list of your keys.
4. Click Create new key and select the JSON key type.
5. Continue with Create. Your JSON service account key will be automatically downloaded to your device.

Data lineage requirements

Dawiso reads table-level and column-level lineage for Google BigQuery from Google’s Data Lineage API, which records how BigQuery jobs move data. To ingest lineage, enable that API and let the service account read it.

Both steps apply per project, and two kinds of project need them:

• Every project you scan: The project whose datasets the data source ingests.
• Every project the BigQuery jobs run in: Google records lineage in the project that runs the job. If your transformations run under a separate compute project, that project needs the API and the role as well.

Enable the Data Lineage API

1. In your Google Cloud console, go to APIs & Services > Enable APIs and services. Make sure you’ve selected the correct project.
2. Search for Data Lineage API.
3. Click Enable.

Using the Google Cloud CLI:

gcloud services enable datalineage.googleapis.com --project <PROJECT_ID>

To verify that the API is enabled:

gcloud services list --enabled --project <PROJECT_ID> | grep datalineage

BigQuery now reports lineage for its jobs automatically. There is nothing to configure per dataset or per table.

Grant the Data Lineage Viewer role

The service account needs the Data Lineage Viewer role (roles/datalineage.viewer) on every project in scope. The role is read-only and grants no access to the data itself.

1. In your Google Cloud console, open the left-side menu, hover over IAM and admin, and click IAM. Make sure you’ve selected the correct project.
2. Find the service account created for Dawiso and click Edit.
3. Click Add another role and select Data Lineage Viewer.
4. Click Save.

Using the Google Cloud CLI:

gcloud projects add-iam-policy-binding <PROJECT_ID> \
    --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" \
    --role="roles/datalineage.viewer"

Repeat the API enablement and this role grant for every project in scope: each project you scan, and each project your BigQuery jobs run in.

A missing API or a missing role costs you no objects. Dawiso ingests every scanned object as usual, records no lineage, and writes a warning to the ingestion log describing what to fix in Google Cloud.

If no lineage appears

Check the following, in this order:

• The Data Lineage API is enabled on that specific project.
• The service account holds roles/datalineage.viewer on that same project.
• The project that runs the BigQuery jobs has both the API and the role, even when it is not the project you scan.
• BigQuery jobs have run since the API was enabled on that project.
• At least 24 hours have passed since those jobs finished.
• The dataset sits in the region the data source is configured with.
• Scan lineage is enabled on the data source. See Create Google BigQuery data source.