The Projects and Datasets fields on a Google BigQuery data source accept more than plain names. This reference covers that pattern syntax, its validation rules, and the safety considerations for an already-ingested data source. For the fields themselves, see Create Google BigQuery data source.
Project patterns
Besides project IDs, Projects accepts patterns:
| Entry | Selects |
|---|---|
my-project | That project. A project ID is always scanned |
regex:<pattern> | Every project the service account can list whose ID matches the pattern |
!regex:<pattern> | Removes the projects it matches from those the regex: entries selected. It never removes a project listed by its ID |
For example, the following value scans sales-prd, and every project whose ID matches analytics-.*-prd, such as analytics-eu-prd and analytics-us-prd, except analytics-sandbox-prd. It doesn’t select analytics-eu-prd-old, because a pattern must match the whole ID:
sales-prd,regex:analytics-.*-prd,!regex:analytics-sandbox-prd- A pattern matches the whole project ID, not a part of it, and ignores case. You don’t need
^and$. - A pattern can’t contain a comma or a semicolon, because those separate the entries. Write
(eu|us)rather than a quantifier such as{2,3}. - Dawiso rejects an invalid pattern when you save the data source.
- A
!regex:entry needs at least oneregex:entry to remove projects from. Otherwise, the ingestion fails.
A pattern can only find projects that Google lists for the service account: projects where the service account holds a role on the project itself, and where the BigQuery API is enabled. A role granted only on a dataset doesn’t make a project visible. See What the service account needs.
The ingestion stops with an error, before anything is removed, when the list of projects can’t be read from Google, or when a regex: entry matches no project.
A project leaves a pattern’s selection when Google stops listing it, for example after the service account loses its role there. An incremental ingestion notices this and stops before it removes anything. A full ingestion has no earlier scan to compare with, so it removes that project’s objects, including user-added content such as descriptions, ownership, or relations. List projects that must always be scanned by their IDs: if the service account can no longer read such a project, the ingestion fails instead.
Dataset qualifiers
- A dataset name filters within every project you scan.
project.datasetselects exactly that dataset, also in a project that isn’t listed in Projects. If Datasets holds only such entries, Dawiso scans exactly those datasets.
A name that matches no dataset in a project is skipped in that project without an error, so check the first ingestion against your list. The ingestion fails when none of the names matches a dataset in any project you scan, or when a project.dataset entry doesn’t exist.