Update Databricks CLI to v1.11.0

[eng-dev-ecosystem-bot]

Update Databricks CLI to v1.11.0

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/vscode

Inputs:

• PR number: 2085
• Commit SHA: c423336f2de0cf946e76615fcde07de445d42ca1

Checks will be approved automatically on success.

[rugpanov]

Executive summary — Databricks CLI v1.10.0 → v1.11.0

CLI

• databricks repos get/update/delete no longer fails with object at path "..." is not a repo for Git-CLI-enabled folders (in preview), which the workspace API reports as directories rather than repos (#6181).
• databricks fs now supports dbfs:/Skills/... paths, routed to the Files API (#6147).

Bundles

Deprecation (the headline item)

• The terraform deployment engine is deprecated and will stop working in a future CLI version. Setting bundle.engine: terraform now emits a deprecation warning; docs cover migrating to the direct engine (#6099).

Direct-engine convergence fixes — all four are "bundle plan/deploy reported a change forever and could never converge":

• webhook_notifications.on_* destinations on jobs, tasks and for_each_task are now compared as unordered sets, so the Jobs API returning them in a different order no longer produces a phantom 1 to change on every run (#6060).
• Pipelines with allow_duplicate_names: true no longer show as a perpetual update — the field is write-only and never returned by the pipelines GET API (#6076).
• Input-only fields (accepted on write, never returned on read — e.g. pipelines' run_as, external locations' skip_validation) are no longer silently dropped from the plan when the new value happens to match the fabricated remote value (#6112).
• An empty grants: [] list no longer plans a spurious create after bundle deployment migrate; emptying a previously deployed list still revokes, then drops the node from state (#6039).
• Reverted RedactiveSenstiveFields (added in #5896, shipped in 1.10.0), which caused permanent drift on the duration field in Postgres resources (#6179).
• Recreating vector_search_indexes no longer fails with "Index ... is currently pending deletion" — the create is retried until the backend releases the name (#6143).

Features / correctness

• Jobs whose ai_runtime_task.code_source_path is a relative local directory now get that directory tarballed (honoring .gitignore and sync.include/sync.exclude), uploaded at deploy, and the path rewritten to the uploaded workspace path (#6110).
• bundle init <template> -o json now reports the files the template wrote, relative to the output directory — so callers passing --output-dir can learn where the template materialized instead of guessing a project-named subdirectory. Default text output unchanged (#6161).
• bundle generate now exports notebooks found inside folders with their extension (notebook.py, not extensionless) (#6144).
• Postgres resource fields documented in the JSON schema (#6164, #6163).

Dependency updates

• databricks-sdk-go v0.165.0 → v0.166.0 (#6175).
• Terraform provider → 1.124.0 (#6174).

What could be integrated into the VS Code extension experience

1. bundle init -o json → replace the glob-based project discovery heuristic (highest value; net-new but small, and it deletes a heuristic)

Today the init wizard runs databricks bundle init --output-dir <parentFolder> interactively in a terminal (BundleInitWizard.ts:175-185) and then has no idea what the template created. It recovers the answer by globbing the parent folder for **/{bundle,databricks}.{yaml,yml} (BundleInitWizard.ts:56 → getSubProjects at BundleFileSet.ts:27-43, pattern at BundleFileSet.ts:12-13). When that glob finds nothing, the user gets a consolation notification telling them to open the folder manually (BundleInitWizard.ts:67-78).

#6161 makes the CLI report exactly which files it wrote, which is the authoritative version of what that glob approximates. Two caveats that make this net-new feature work, not a flag flip:

• The interactive path deliberately forces text output — getBundleInitEnvVars hardcodes DATABRICKS_OUTPUT_FORMAT: "text" (CliWrapper.ts:605) because a human reads that terminal. Getting JSON out of a terminal-hosted wizard means capturing it to a temp file rather than just adding -o json, so this is real work.
• The non-interactive path is the cheap one: CliWrapper.bundleInit() (CliWrapper.ts:671-692) already runs bundle init via execFile with a --config-file and captures stdout. It's used for the legacy-project migration (BundleProjectManager.ts:324-329), which writes a template containing databricks.yml.tmpl (resources/migration-template/template/databricks.yml.tmpl) into a known directory. Adding --output/-o json there is a few lines and would let the migration verify what it wrote instead of assuming success (BundleProjectManager.ts:330 currently just logs "Successfully finished bundle migration").

Worth noting the glob has a real failure mode this fixes: it filters out the root itself (BundleFileSet.ts:40-42), so a template that materializes directly into the chosen parent folder is invisible to discovery and always lands in the "we haven't detected any Databricks projects" branch.

2. Terraform-engine deprecation — audit the bundled-terraform machinery (cleanup + a planning question; the most consequential item here)

This is the one item with real strategic weight, because the extension ships and pins terraform itself:

• scripts/setupCLIDependencies.ts asks the bundled CLI for its terraform dependency metadata (bundle debug terraform --output json, lines 39-52), downloads the matching terraform binary with SHA verification and the Databricks provider zip, and writes a terraformMetadata block into package.json (lines 100-111).
• scripts/package-vsix.sh:55-58 runs that for every arch except win32-arm64.
• At runtime getCLIDependenciesEnvVars exports DATABRICKS_TF_VERSION, DATABRICKS_TF_EXEC_PATH, DATABRICKS_TF_PROVIDER_VERSION and DATABRICKS_TF_CLI_CONFIG_FILE (envVarGenerators.ts:171-187), and extension.ts:401-420 pushes them into context.environmentVariableCollection so they apply to every terminal, then writes a filesystem-mirror .tfrc (terraformUtils.ts:29-43).
• bundleSummarise passes --force-pull specifically to "regenerate local terraform state ... so the provider version used in the local state matches the bundled version we supply with the extension" (CliWrapper.ts:571-579).

Two grounded observations, no roadmap speculation:

• No-op for this PR: the extension never sets bundle.engine, so nothing here newly emits the #6099 warning. I grepped for it — the only engine hits under src/ are runtime_engine in the generated BundleSchema.ts (lines 642, 1682, 5844, 6556) plus an unrelated comment; there is no bundle.engine write anywhere. Users who set it themselves in databricks.yml will see the warning in the "Databricks Bundle Logs" channel, since runBundleCommand routes CLI stderr to the logger (CliWrapper.ts:246-252, CliWrapper.ts:282) and deploy force-opens that channel (CliWrapper.ts:703).
• Real follow-up, not for this PR: once terraform stops working, the entire download/pin/env-var pipeline above becomes dead weight in every VSIX, and the --force-pull rationale at CliWrapper.ts:576-578 stops being a reason. That is a deliberate cleanup PR (build script + env vars + packaging + the --force-pull comment), and this bump is the signal to start tracking it.

3. Direct-engine convergence fixes — free win for the resource explorer, zero code change

These four fixes (#6060, #6076, #6112, #6039) land directly in extension UI with no work at all, which is worth calling out explicitly in the PR description as the user-visible payoff.

The resource explorer reads modified_status off each resource in the bundle summary output (BundleRemoteStateModel.ts:16-34) and renders it as a git-style badge — C/D/U with themed colors and an "Updated after deploy" tooltip (DecorationUtils.ts:5-33), consumed by JobTreeNode.ts:66-75, PipelineTreeNode.ts:66-75 and UnknownResourceTreeNode.ts:58-62, and also feeding context values (ContextUtils.ts:16, 29) and collapsible state (DecorationUtils.ts:35-48).

So a phantom diff in the CLI was a permanently stuck U badge in the tree: a job with reordered webhook_notifications, or a pipeline with allow_duplicate_names: true, showed "Updated after deploy" forever no matter how many times the user deployed. Those two cases are the most likely to bite extension users specifically, since jobs and pipelines are the two resource types with dedicated tree nodes. The Postgres-drift revert (#6179) fixes the same class of stuck badge for Postgres resources, though those render through the generic UnknownResourceTreeNode — ResourcesClass in the generated schema (BundleSchema.ts:282-337) has no Postgres key.

4. Explicit no-ops for the extension

Calling these out so reviewers don't go looking for integration work:

• dbfs:/Skills/... in databricks fs (#6147) — the extension never shells out to databricks fs. The complete set of subcommands it invokes is bundle sync|schema|validate|summary|init|deploy|destroy|run (CliWrapper.ts:382, 496, 544, 571, 680, 708, 746, 776, 820), auth profiles (CliWrapper.ts:399-400), and environments setup-local (setupLocalArgs.ts:33). No fs.
• databricks repos fix (#6181) — no-op for the same reason: the extension's only repo operation goes through the SDK, not the CLI (wsClient.repos.create(...) at WorkspaceFsCommands.ts:283-287), and the /Repos references in WorkspaceFsDataProvider.ts:153-156 are just tree-sort priority paths. The CLI-side get/update/delete fix is unreachable from here. (SDK-side behavior would move with @databricks/sdk-experimental at package.json:1579, not with this bump — the Go SDK bump in #6175 only affects the CLI binary.)
• bundle generate notebook extensions (#6144) — the extension never calls bundle generate; I grepped and there is no such invocation.
• ai_runtime_task tarball packaging (#6110) — transparent. It happens inside bundle deploy, which the extension already invokes as an opaque subprocess (CliWrapper.ts:694-729), so users get it for free. There is no extension-side awareness to add: ai_runtime_task does not appear anywhere in src/, including the generated BundleSchema.ts (0 matches).
• Postgres JSON-schema docs (#6164, #6163) — these flow into YAML autocomplete automatically at runtime, no regeneration needed: registerBundleAutocompleteProvider calls cli.getBundleSchema() (bundleAutocompleteProvider.ts:13), which shells out to bundle schema (CliWrapper.ts:494-501) and hands the result to redhat.vscode-yaml (bundleAutocompleteProvider.ts:44-62). Users get the new field docs on hover the moment the new binary ships.

One thing worth flagging independently of this bump: the checked-in src/bundle/BundleSchema.ts is generated by scripts/writeBundleSchema.ts via yarn package:bundle-schema:write (package.json:1555), but its last content change was 950051d/62a94e1 (the DABs rename, several CLI versions ago). It's stale relative to the pinned CLI — which is harmless for autocomplete (that path is runtime-generated, above) but means the TypeScript types used by BundleRemoteStateModel and the tree nodes lag the CLI's actual schema. Regenerating it is out of scope for a version bump; the previous bump 3f639a1 touched only package.json:1539, and matching that keeps this PR reviewable.

_{🤖 Auto-generated executive summary of the CLI v1.10.0 → v1.11.0 changelog. Integration notes are opportunities, not commitments — verify before acting.}