Skip to content

Centralize BigQuery Avro loads in BqUtils#715

Open
kmontemayor2-sc wants to merge 4 commits into
mainfrom
kmonte/bq-utils-load-files
Open

Centralize BigQuery Avro loads in BqUtils#715
kmontemayor2-sc wants to merge 4 commits into
mainfrom
kmonte/bq-utils-load-files

Conversation

@kmontemayor2-sc

Copy link
Copy Markdown
Collaborator

What

  • Adds BqUtils.load_files_to_bq for loading one or more GCS files with a caller-provided BigQuery job configuration.
  • Preserves synchronous and asynchronous load behavior while centralizing load timing and result logging.
  • Routes embedding and prediction Avro exports through BqUtils and uses BqUtils.join_path for the destination table.
  • Adds focused unit coverage for the utility and export delegation, plus end-to-end prediction export coverage.

Why

The export helpers maintained their own BigQuery client and duplicated load-job orchestration. Centralizing that behavior in BqUtils gives file-based loads the same client ownership and a reusable sync/async API while preserving existing public export signatures and load semantics.

Testing

  • bq_test.py: 9 tests passed.
  • Export delegation tests: 4 tests passed.
  • Scoped ty checks passed for all changed Python files.
  • Ruff check and formatting passed for all changed Python files.

@mkolodner-sc mkolodner-sc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Kyle!

@kmontemayor2-sc
kmontemayor2-sc marked this pull request as ready for review July 22, 2026 14:00
@kmontemayor2-sc
kmontemayor2-sc added this pull request to the merge queue Jul 22, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 22, 2026
@kmontemayor2-sc
kmontemayor2-sc added this pull request to the merge queue Jul 22, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jul 22, 2026
@kmontemayor2-sc
kmontemayor2-sc added this pull request to the merge queue Jul 22, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants