Add 'execution_project' option to model configuration for BigQuery adapter #1455
+748
−3
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
related to #343 / #226
Problem
BigQuery model execution is defined at the
profiles.yml. This PR introduces a per-resourceexecution_projectconfiguration that lets each model explicitly choose which GCP project to use for query execution at run time. This can be useful for when the model selector is used for multiple models; perhaps one model in that grouping is a known, heavy model that needs to utilize a BigQuery project with more slots available. This allows the user to have a more fine-grained control over which BigQuery project a specific model is run in.The default behavior is not changed and the execution project specified in the
profiles.ymlis used by default.This solution provides a similar result as the Snowflake connector configuration
Solution
The
BigQueryConnectionManagernow maintains a cache of BigQuery clients keyed by execution_project, protected by a lock. Each dbt thread records which execution project it’s currently using so we can restore it later.pre_model_hookreads config.execution_project, which can be either a plain string or a{target-name: project-id}mapping. If the configuration specifies a different project than the profile’s default, the adapter calls switch_execution_project, which:post_model_hookthen restores the previousexecution_projectreturned bypre_model_hook, ensuring subsequent models don’t inherit the override.Cached clients are marked so they aren’t closed prematurely, and
cleanup_allnow properly closes everything in the cache.Checklist