fix: replace instead of drop when rebuilding a managed Iceberg table

[cemsbr]

Resolves #1662

Description

Stacked on #1669 — the first three commits are that PR's; this PR adds one commit on top and will be rebased onto main once #1669 merges.

#1662 reports the drop-then-create on --full-refresh of an incremental model on a Unity Catalog managed Iceberg table. The table materialization has the same gate and hits it on every run:

not (existing_relation.can_be_replaced and adapter.resolve_file_format(config) in ('delta', 'iceberg'))

With use_managed_iceberg on and table_format: iceberg, resolve_file_format returns parquet, so the existing table is always dropped before a create or replace table ... using iceberg CTAS that would have been atomic on its own. The table is unavailable for the whole CTAS (minutes on large tables) and concurrent readers fail with TABLE_OR_VIEW_NOT_FOUND; table history/time travel restarts on every run.

This change reuses format_allows_create_or_replace(catalog_relation, existing_relation) from #1669 in both the v1 and v2 branches of table.sql. The is_shallow_clone and type != 'table' arms are unchanged, and Delta targets keep the same behavior as before (catalog_relation.file_format == 'delta' and existing_relation.is_delta).

Not included: materializations/seeds/helpers.sql (databricks__reset_csv_table) has the same gate; happy to fold it in here or in a follow-up, whichever you prefer.

Behavior change

When the existing table's provider differs from the target (an existing managed Iceberg table with a Delta target, e.g. a project turning use_managed_iceberg back off), the old gate attempted create or replace table across providers; Databricks rejects that with [MANAGED_ICEBERG_OPERATION_NOT_SUPPORTED] Managed Iceberg tables do not support REPLACE with different providers. and the rebuild failed. With the shared predicate that case drops and recreates the table instead, matching #1669's test_delta_target_on_iceberg_relation. Probe results in the comments below.

Checklist

• I have run this code in development and it appears to resolve the stated issue — as a root-project override of the table materialization with the same predicate, on dbt-databricks 1.12.5 against a UC managed Iceberg table (second run no longer emits Applying DROP; information_schema.tables.created is preserved)
• This PR includes tests, or tests are not required/relevant for this PR — TestManagedIcebergTableRebuild in tests/functional/adapter/iceberg/test_iceberg_support.py, mirroring fix: replace instead of drop on managed Iceberg full refresh #1669's TestManagedIcebergFullRefresh; needs a workspace, so it relies on /integration-test
• I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.
• [Optional] I have run the dbt-databricks-pr-ready project skill for this PR and addressed its merge-readiness feedback

[moomindani]

Thanks for picking this up, and for reusing the macro and test shape from #1669.

I ran your branch here against a serverless SQL warehouse, since fork PRs can't start CI on their own: code-quality clean, unit 1349 passed / 4 skipped, tests/functional/adapter/iceberg/ 12 passed. Reverting only table.sql makes TestManagedIcebergTableRebuild fail, so the new test does catch it. I also checked the v1 path is fine to stop dropping — databricks__create_table_as already emits create or replace for a managed Iceberg model.

One question. The old gate looked only at the target format, and format_allows_create_or_replace also requires the existing relation's provider to match, so an existing Iceberg table with a Delta target — a project turning use_managed_iceberg back off — moves from replace to drop. Dropping seems the safer of the two, but was that intentional?

Since this is stacked on #1669, that one needs to land first; it's waiting on a maintainer to approve its workflows.

[cemsbr]

Thanks for running it end to end, and for the question — it made me check rather than assume.

Yes, intentional. Reusing the predicate unchanged keeps table and incremental behaving the same on a rebuild, and the "provider changed → drop" arm is the one #1669 already encodes in test_delta_target_on_iceberg_relation. But I did not know what the old gate's REPLACE actually did across providers, so I probed it on a Unity Catalog serverless SQL warehouse (dbt-databricks 1.12.5):

existing table	statement	result
using iceberg (managed)	create or replace table t using delta as select ...	[MANAGED_ICEBERG_OPERATION_NOT_SUPPORTED] Managed Iceberg tables do not support REPLACE with different providers. — table left intact, still Provider = iceberg
using delta	create or replace table t using iceberg as select ...	same error — table left intact, still Provider = delta

So in the case you describe (existing Iceberg table, project turns use_managed_iceberg back off), the old gate did not replace: the rebuild failed with that error until someone dropped the table by hand. Dropping first is the only path that works, so I would keep the predicate as it is for this PR.

One thing the probe surfaced about the predicate itself, which is #1669's territory and your call: the Delta arm (catalog_relation.file_format == 'delta' and existing_relation.is_delta) is also true for a managed-Iceberg target over a legacy Delta table (test_managed_iceberg_target_on_delta_relation → replace). Per the second row above that REPLACE fails with the same error, so a project flipping the flag on over existing Delta tables would hit it on the first full refresh (and, with this PR, on the first table rebuild). Making that arm symmetric — existing_relation.is_delta only when the target is not managed Iceberg — would turn that into a drop-then-create as well. Happy to send that as a follow-up on top of whichever lands first, or to fold it here if you prefer.

I've added a short "Behavior change" note to the PR description.

[moomindani]

Your read was right, and thank you for probing it rather than assuming — I had the same two rows before pushing, so we agree on the facts:

existing	statement	result
using delta	create or replace ... using iceberg	MANAGED_ICEBERG_OPERATION_NOT_SUPPORTED, table left as delta
using iceberg	create or replace ... using delta	same, table left as iceberg

On your two points:

The "provider changed → drop" arm is intentional — agreed, keep it. Your probe settles what I was actually unsure about: the old gate never replaced successfully in that case, it just failed until someone dropped the table by hand. So this is not a behavior change worth hedging over; it turns an error into a working rebuild.

The asymmetric Delta arm was a real bug, and I've folded the fix into #1669 rather than take you up on the follow-up — the predicate lives there, and leaving a known-broken arm in the commit that introduces it didn't seem right. I'd written the symmetric version first and then talked myself out of it on "don't change behavior beyond the reported bug", which was the wrong call: the behavior I preserved was one I hadn't verified.

1024e315 on #1669 makes the arms mutually exclusive, flips test_managed_iceberg_target_on_delta_relation to False with the reason in the docstring, and adds a functional test (TestManagedIcebergOverExistingDelta) that creates the table as Delta, turns the flag on, and runs --full-refresh. On the old predicate that test reproduces your second row as a dbt error; on the new one it passes with Provider = iceberg. Unit 1357 passed, code-quality clean, tests/functional/adapter/iceberg/ 12 passed. The commit message credits you for the probe.

I also rebased #1669 onto main, so the stack needs redoing — but table.sql shouldn't need any edit, since the corrected predicate is the same shared macro your change already calls. Your table path picks up the flag-on-over-Delta case for free.