Filter catalog text_only_columns to existing columns to prevent agate warnings #1437
+24
−11
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
physcat
force-pushed
the
fix-bq-table-owner-warning
branch
from
b3baa3f to
20794e8
Compare
physcat
force-pushed
the
fix-bq-table-owner-warning
branch
from
20794e8 to
cdabad2
Compare
… warnings BigQuery's catalog macro does not return a 'table_owner' column, but the base adapter implementation was attempting to force this column to text type. This caused agate to emit RuntimeWarnings during 'dbt docs generate': "table_owner" does not match the name of any column in this table. This change makes the implementation more defensive by filtering the text_only_columns list to only include columns that actually exist in the table. This prevents warnings when adapters return different metadata columns while still ensuring all returned columns are properly typed as text.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Resolves #1135
When running
dbt docs generate, agate emits RuntimeWarnings for columns specified intext_only_columnsthat don't exist in the catalog results:This happens because some adapter implementations (e.g., BigQuery, Athena) don't return all the metadata columns that were added to the
text_only_columnslist in PR #1057, specifically thetable_ownercolumn.Solution
This PR takes a defensive approach by filtering the
text_only_columnslist at runtime to only include columns that actually exist in the returned catalog data. This prevents warnings when adapters return different metadata columns while still ensuring all returned columns are properly typed as text.Alternative Approach Note: I want to acknowledge PR #1404 by @morgan-dgk which addresses the same issue by removing
table_ownerfrom the base implementation and moving it to adapter-specific implementations. That's a valid architectural approach, and since that PR is already in progress, I'm happy to close this PR if the team prefers that solution.The key difference between the two approaches:
Checklist