Skip to content

Conversation

@bordway-shop
Copy link

@bordway-shop bordway-shop commented Nov 19, 2025

related to #343 / #226

Problem

BigQuery model execution is defined at the profiles.yml. This PR introduces a per-resource execution_project configuration that lets each model explicitly choose which GCP project to use for query execution at run time. This can be useful for when the model selector is used for multiple models; perhaps one model in that grouping is a known, heavy model that needs to utilize a BigQuery project with more slots available. This allows the user to have a more fine-grained control over which BigQuery project a specific model is run in.

The default behavior is not changed and the execution project specified in the profiles.yml is used by default.

This solution provides a similar result as the Snowflake connector configuration

Solution

The BigQueryConnectionManager now maintains a cache of BigQuery clients keyed by execution_project, protected by a lock. Each dbt thread records which execution project it’s currently using so we can restore it later.

pre_model_hook reads config.execution_project, which can be either a plain string or a {target-name: project-id} mapping. If the configuration specifies a different project than the profile’s default, the adapter calls switch_execution_project, which:

  1. Either reuses a cached client for that project or creates one on demand then
  2. Swaps the thread’s connection handle to point at the new client.
  3. post_model_hook then restores the previous execution_project returned by pre_model_hook, ensuring subsequent models don’t inherit the override.

Cached clients are marked so they aren’t closed prematurely, and cleanup_all now properly closes everything in the cache.

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

@bordway-shop bordway-shop requested a review from a team as a code owner November 19, 2025 21:52
@cla-bot
Copy link

cla-bot bot commented Nov 19, 2025

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @bordway-shop

@cla-bot cla-bot bot added the cla:yes The PR author has signed the CLA label Nov 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla:yes The PR author has signed the CLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant