perf: avoid re-applying unchanged column tags on retained relations

[sd-db]

Describe the problem

Column-level databricks_tags are emitted as one ALTER ... ALTER COLUMN ... SET TAGS statement per tagged column. Several materialization paths replay the full configured column-tag map, even on reruns where the relation may retain its existing column tags. With many tagged columns, these writes can dominate model runtime.

Column tags are currently applied by the table, incremental, view, materialized_view, streaming_table, and snapshot materializations.

Current reconciliation is inconsistent:

• Ordinary incremental reruns and the View ALTER path consume ColumnTagsConfig through a relation changeset. perf: emit only changed column databricks_tags keys in ALTER COLUMN SET TAGS #1668 narrows that diff to new or changed tag keys.
• Table in-place replacement is currently handled together with relation-level tags in perf: apply only new or changed databricks_tags on table re-runs #1572.
• Incremental full refresh, Snapshot reruns, View replacement, and Materialized View or Streaming Table build paths call apply_column_tags with the full model configuration.
• Materialized View and Streaming Table relation configs do not currently include ColumnTagsProcessor, so column tags are not part of their configuration changesets.

This should be tracked independently from relation-level databricks_tags, because the preservation and update behavior is per column and can differ by materialization lifecycle.

Expected behavior

For each supported materialization and lifecycle operation:

1. Verify through server-observable functional tests whether the operation preserves existing column tags.
2. When tags are preserved, fetch column-tag metadata only when column tags are configured and apply only new or changed keys.
3. When the relation is newly created or dropped and recreated, apply the full desired column-tag map without fetching stale metadata.
4. Preserve remote-only tags under the existing set-only reconciliation semantics.
5. Ensure an unchanged rerun emits no column-tag writes when preservation is guaranteed.

The test matrix should cover:

• Table V1 and V2 in-place replacement: coordinate with perf: apply only new or changed databricks_tags on table re-runs #1572.
• Incremental ordinary rerun and in-place full refresh.
• Snapshot rerun of the retained target table.
• View replace and view_update_via_alter paths.
• Materialized View refresh, alter, and replace paths.
• Streaming Table refresh, alter, and replace paths.

View, Materialized View, and Streaming Table optimization should only be enabled after tests prove that the relevant Databricks operation retains column tags. Existing tests reapply the configured tags after lifecycle operations, so they do not establish preservation by themselves.

Fresh creates and drop-and-recreate paths are explicitly out of the optimization: they must continue to apply all configured column tags.

Related

• Databricks_tags are set every run #1308
• perf: apply only new or changed databricks_tags on table re-runs #1572
• perf: emit only changed column databricks_tags keys in ALTER COLUMN SET TAGS #1668
• Column tags are not added for snapshots #1441
• View column tags not applied on the ALTER update path (view_update_via_alter: true) → "main is not being called" error #1525