Skip to content

persist_docs.columns triggers a full-plan probe query on views even when the model has no column docs #1680

Description

@PhilippLange

Describe the bug

With persist_docs: {columns: true}, databricks__create_view_as always calls get_columns_in_query(sql) before creating a view, which executes select * from (<full model sql>) as __dbt_sbq where false limit 0 against the warehouse. It does this even when the model has no columns: entries in any schema YAML, in which case the resulting column list contains no comment clauses and is semantically identical to emitting no column list at all.

On a wide view the probe hangs for a long time. where false is only folded after full analysis, so Spark still plans the whole query and does Delta metadata/file listing on every source.

Expected behavior

No probe query when the model has no column descriptions. dbt's own macros/adapters/persist_docs.sql already uses this guard:

{% if for_columns and config.persist_column_docs() and model.columns %}

The Databricks view path omits the and model.columns short-circuit.

Suggested fix

In dbt/include/databricks/macros/relations/view/create.sql:

{%- if config.persist_column_docs() and model.columns -%}

Steps to reproduce

  1. Set persist_docs: {relation: true, columns: true} in dbt_project.yml
  2. Create a materialized='view' model with no corresponding schema.yml entry
  3. Run dbt run -s that_model and observe the where false limit 0 query in the Databricks query history

Notes

  • Workaround: persist_docs={'relation': True, 'columns': False} per model.

System information

  • dbt-databricks: 1.11.6
  • dbt-core: 1.11.6
  • databricks-sql-connector: 4.1.3

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions