Package a local directory code_source_path for AI Runtime tasks - #6110
Merged
Conversation
Let ai_runtime_task.code_source_path point at a local directory, not just a pre-built tarball. A new aicode mutator (run in the build phase, before libraries.ExpandGlobReferences/ReplaceWithRemotePath) detects a local-directory value, packages it into a reproducible, content-addressed tarball — honoring .gitignore and the top-level sync.include/exclude globs — uploads it to the user's ~/.air/repo_snapshots directory, and rewrites code_source_path to the uploaded (de-/Workspace-prefixed) path. Because it runs first, PR #5922's artifact-style collection then sees an already-remote path and skips it, so the two compose: a directory is packaged by the CLI, a pre-built .tgz still flows through the artifact path. Also synthesizes a requirements.yaml next to the task's command_path from the job's serverless environments[] spec, so the AI Runtime harness sets up the workload environment. command_path translation itself is provided by #5922. Verified end-to-end on a live workspace: a plain bundle deploy of a directory code_source_path yields a job whose run terminates SUCCESS; sync.exclude and .gitignore entries are absent from the uploaded snapshot; unchanged code skips re-upload. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: 215d43d
8 interesting tests: 4 RECOVERED, 4 SKIP
Top 6 slowest tests (at least 2 minutes):
|
The aicode mutator packages a local directory code_source_path at deploy. A
pre-built tarball delivered via an `artifacts` block (code_source_path pointing
at a local *file*) must instead flow through the standard artifact-upload path,
exactly as the two-path design intends. Validate rejected such a path outright
("code_source_path not found" — the artifact tarball does not exist yet at
initialize time), breaking the existing bundle/artifacts/ai_runtime_code_source
acceptance test.
Both Validate and collectLocalCodeSources now skip any local code_source_path
that does not resolve to an existing directory, leaving it for the artifact
uploader. The packaging-specific constraints (git_source, immutable_folder) only
apply once the path is confirmed to be a directory this mutator will package.
Co-authored-by: Isaac
- Remove the never-set `client` filer field from packageAndUpload and synthesizeRequirements (only ever nil); build the workspace filer directly. - Read the current user directly (resolved in the initialize phase) instead of the defensive nil-guarded helper. - Surface a stat error other than not-exist (e.g. unreadable directory) instead of silently treating it as "not a directory to package"; shared isExistingDir helper used by collect and Validate. - Pass GitSource into validateTask (nil-check inside) to keep validation logic contained. - Collect a single direct-task code_source_path pattern, matching Validate and SynthesizeRequirements (for_each_task is unsupported until all three gain it). - SynthesizeRequirements writes requirements.yaml next to every deployment's command_path (deduped by directory), not just the first. Co-authored-by: Isaac
Reworks the local_code_source acceptance test per review: - Assert the uploaded tarball's contents via list_code_snapshot.py: gitignored files (ignored_by_git.txt, data/) are excluded, *.log is excluded via sync.exclude, and data/model.bin is force-included via sync.include. - Use print_requests.py instead of hand-rolled jq (testing.md rule); extend its --del-body to also strip the top-level raw_body so the binary tarball upload doesn't dump into the golden. - Add a cache-miss case: editing a file changes the content hash and re-uploads. Co-authored-by: Isaac
print_requests.py path filters and the workspace export in list_code_snapshot.py were rewritten by Git Bash's MSYS path conversion on Windows (e.g. /repo_snapshots/ -> C:/Program Files/Git/repo_snapshots/), failing the test. Use // leading filters (matching the existing convention) and set MSYS_NO_PATHCONV for the export. Co-authored-by: Isaac
Split subprocess.run kwargs onto separate lines to satisfy ruff format. Co-authored-by: Isaac
Reworks PackageAndUpload to place the code snapshot inside the bundle instead of uploading it to a shared ~/.air/repo_snapshots workspace cache. The mutator now writes the content-addressed tarball into the bundle's sync tree (.air_snapshots/) and rewrites code_source_path to the workspace path it will occupy once synced; it performs no workspace write itself. Normal bundle file sync uploads it during the deploy phase. Why: - Build phase must not mutate the workspace (it runs before `bundle plan`). The old filer upload violated that; now the only build-phase work is local prepare-and-archive, and the upload happens in the deploy phase via file sync. - No deploy-time side-effect outside the bundle: `bundle destroy` removes the snapshot like any other bundle file, so the home-dir cache can no longer compound across bundles/deploys. - Dedup is preserved: the name is content-addressed and bundle file sync is incremental, so unchanged code keeps the same synced path and is not re-uploaded. Tradeoff: the cache is now per-bundle rather than shared across bundles. See the package doc for the (rejected) TranslatePaths alternative and why archiving stays in the build phase. Co-authored-by: Isaac
The snapshot is now injected into the bundle sync root as an in-memory overlay file rather than written under the working tree. This keeps the deploy artifact inside the bundle (synced to <root>/files/.air_snapshots/, removed by `bundle destroy`, content-addressed so unchanged code is not re-uploaded) while leaving the user's working tree clean — no generated .tar.gz appears in their checkout. Adds libs/vfs.Overlay: a vfs.Path wrapper that serves a set of in-memory files in addition to a base path, participating in Open/Stat/ReadDir/ReadFile and fs.WalkDir so bundle file sync uploads the overlaid files transparently. PackageAndUpload builds each snapshot in memory and swaps b.SyncRoot for an overlay carrying them. Co-authored-by: Isaac
ben-hansen-db
self-requested a review
July 31, 2026 23:19
ben-hansen-db
left a comment
ben-hansen-db
left a comment
Contributor
There was a problem hiding this comment.
code_source_path: ../shared # points outside the bundle
Error: failed to list files for code_source_path "../shared": stat ../shared: invalid argument
can we make this error better?
Fixes from PR review: - Reject a code_source_path that escapes the bundle sync root (e.g. "../shared") in Validate with a clear message, instead of failing later as an opaque io/fs "invalid argument". - Reject a local code_source_path combined with source-linked deployment: that mode makes files.Upload a no-op and ignores workspace.file_path, so the packaged snapshot would never be uploaded and the job would point at a missing path. - Reject a local code_source_path nested under a for_each_task, which the mutator does not package (it operates on direct tasks only) — previously silently skipped. - Error when the code_source directory has no files to package (all excluded by .gitignore/sync.exclude, or empty) rather than deploying a job with no code. - Force-include the .air_snapshots snapshot dir in GetSyncIncludePatterns (like .databricks) so a user ignore rule such as "*.tar.gz" can't filter the deployed job's code_source_path archive out of the sync set. Shared bundle.AiCodeSnapshotDir constant keeps the mutator and the sync include in agreement. Co-authored-by: Isaac
Remove the SynthesizeRequirements mutator. It wrote a co-located requirements.yaml next to command_path during the build phase — which (a) made workspace writes in a phase that runs before `bundle plan` and (b) duplicated dependency delivery that the native path already handles. The AI Runtime runtime installs pip deps from the job's serverless environment (environments[].spec.dependencies) via the server-side --deps-config path; the co-located requirements.yaml is the legacy mechanism new clients no longer create. convert-to-dabs already emits deps into environments[].spec, so the sidecar was redundant. Verified end-to-end on A10: with SynthesizeRequirements disabled and no requirements.yaml uploaded, the run logs "No co-located requirements.yaml ... skipping" and installs torch from the inline environment deps to SUCCESS. Co-authored-by: Isaac
Second review round: - Reject a sync.exclude that matches the reserved .air_snapshots dir. The sync set is (git ∪ include) − exclude, so exclude wins over the force-include; a pattern like ".air_snapshots/*" or "**/*.tar.gz" would silently drop the generated archive and leave the job pointing at an un-uploaded path. Validate now errors. - Reject a real file/dir at .air_snapshots in the bundle: it would collide with the in-memory overlay (sync would carry the user's entry, not the generated archive). - Both guards only fire when a task actually packages a local code_source. - Rename PackageAndUpload -> PackageCodeSource: it packages and overlays the archive for file sync to upload; it no longer uploads itself. - Acceptance test: add a second AI Runtime task so two tarballs are packaged, both under the repo-root .air_snapshots; assert a no-op re-deploy; and destroy at the end so the test cleans up after itself. list_code_snapshot.py now lists every task's archive. Co-authored-by: Isaac
Close two review gaps:
- Assert a literal `bundle plan` no-op ("0 to add, 0 to change, 0 to delete")
after deploy, instead of a no-op re-deploy proxy.
- Add an empty_code_source acceptance test: a code_source dir whose contents are
all filtered out (src/.gitignore of "*") makes `bundle deploy` fail with the
"no files to package" error, covering packageOne's empty-source unhappy path
end-to-end (the pipeline needs a workspace client, so this is exercised via
acceptance rather than a mocked unit test).
Co-authored-by: Isaac
Only force-sync .air_snapshots for bundles that actually package an AI Runtime code_source, rather than for every bundle. The aicode mutator sets a new Bundle.HasAiRuntimeCodeSnapshot flag when it packages one, and GetSyncIncludePatterns gates the force-include on it. Keeps the generic sync include list from carrying a feature-specific entry for unrelated bundles. Also tighten the comments added across this feature to be concise. Co-authored-by: Isaac
The fragment described the pre-review behavior. requirements.yaml is no longer synthesized (deps come from the environments spec), and the archive is packaged into the bundle and uploaded by file sync, not written to a separate workspace cache dir. Co-authored-by: Isaac
ben-hansen-db
approved these changes
Aug 3, 2026
ben-hansen-db
left a comment
ben-hansen-db
left a comment
Contributor
There was a problem hiding this comment.
Looks good but I think mode should be no more restrictive than whatever user has locally
The snapshot builder hardcoded 0o644, stripping the execute bit — a bundled helper the user invokes (e.g. a ./run.sh called from command.sh) would arrive non-executable and fail at runtime. Preserve the owner execute bit (0o755 when set, else 0o644) while still normalizing the rest of the mode. Deriving the mode from the file's own bits keeps the archive reproducible. Co-authored-by: Isaac
Windows file modes don't carry a Unix execute bit, so writing 0o755 doesn't make the file report executable and the archived-mode assertion fails there. The bit only matters on the Unix hosts that run the workload; skip the test on Windows. The production path is already Windows-safe (files report non-exec -> 0o644). Co-authored-by: Isaac
…erlay paths - validate: gate the snapshot-dir guards on an existing local *directory*, not any local path. A pre-built tarball (a local file, the #5922 flow) is uploaded by the artifact path and packages no snapshot, so a natural `sync.exclude` of "*.tar.gz" was wrongly rejected. Guarded in the file-based acceptance test. - codeSourceFiles: drop entries outside the code directory. sync.include is force- added regardless of the scoped walk, so an unrelated include made an all-filtered directory look non-empty and shipped an empty archive. The empty_code_source test now covers that shape and uses musterr. - vfs.Overlay: reject names that aren't relative in-root files. An absolute name hung the ancestor walk (path.Dir("/") is never "."), and "" resolved to the root. - print_requests.py: add --del-field for top-level request fields instead of overloading --del-body, which only edits the parsed JSON body. - Rename package_upload.go to package_code_source.go (it performs no upload), trim the changelog fragment, and document that entry modes are not cross-platform identical (Windows has no execute bit). Co-authored-by: Isaac
Co-authored-by: Isaac
janniklasrose
left a comment
janniklasrose
left a comment
Contributor
There was a problem hiding this comment.
couple of small changes on the newly created list_code_snapshot, otherwise lgtm & ready to merge
| @@ -0,0 +1,69 @@ | |||
| #!/usr/bin/env python3 | |||
Contributor
There was a problem hiding this comment.
use main() for this script
| if pos >= len(text): | ||
| break | ||
| obj, pos = decoder.raw_decode(text, pos) | ||
| requests.append(obj) |
Reuse read_json_many from print_requests instead of re-implementing the concatenated-JSON decode. Co-authored-by: Isaac
janniklasrose
approved these changes
Aug 4, 2026
Sankalp-Mittal
added a commit
that referenced
this pull request
Aug 5, 2026
The local_code_source test filtered recorded uploads by URL path, which no longer matches: /workspace/import carries the target filename in the multipart body. Filter on the body instead, the same way auto-migrate-empty-tfstate does. --del-field raw_body is dropped because the tarball is binary and the request recorder already summarizes it as a size placeholder. This test was added in #6110, after the upload migration branch was cut, so it was not covered by the earlier fixture updates. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: da9949b
412 interesting tests: 385 MISS, 20 FAIL, 4 RECOVERED, 2 KNOWN, 1 SKIP
Top 50 slowest tests (at least 2 minutes):
|
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Aug 6, 2026
## Release v1.11.0 ### CLI * Fixed `databricks repos get/update/delete` failing with `object at path "..." is not a repo` for Git-CLI-enabled folders (currently in preview), which the workspace API reports as directories rather than repos ([#6181](#6181)). * Support `dbfs:/Skills/...` paths in `databricks fs` commands, routed to the Files API. ([#6147](#6147)) ### Bundles * For jobs where `ai_runtime_task.code_source_path` is a relative path to a local directory, the directory is now packaged into a tarball (honoring `.gitignore` and `sync.include`/`sync.exclude`), uploaded during deployment, and `code_source_path` is rewritten to the uploaded workspace path. ([#6110](#6110)) * Added JSON output to `bundle init`. Running `databricks bundle init <template> -o json` now reports the files the template wrote, relative to the output directory. This lets callers that pass `--output-dir` learn where the template materialized instead of assuming the output is a single directory named after the project. The default text output is unchanged. ([#6161](#6161)) * The terraform deployment engine is deprecated and will stop working in a future version of the CLI. Setting `bundle.engine: terraform` now emits a deprecation warning. See https://docs.databricks.com/aws/en/dev-tools/bundles/direct for how to migrate to the direct deployment engine. ([#6099](#6099)) * Fixed the direct deployment engine planning a spurious `create` for an empty `grants: []` list. Terraform records no grants resource for such a list, so `bundle plan` after `bundle deployment migrate` no longer reports an action for it. Emptying a previously deployed list still revokes the grants, after which the node is dropped from the deployment state instead of being reported as unchanged forever. ([#6039](#6039)) * Fixed `bundle generate` downloading notebooks found inside a folder without their file extension. They are now exported like top-level notebooks, so a Python notebook lands as `notebook.py` instead of an extensionless file ([#6144](#6144)). * direct: `webhook_notifications.on_*` destinations on jobs, tasks, and `for_each_task` are now compared as unordered sets. Previously the Jobs API returning these lists in a different order than submitted produced a phantom diff that `bundle plan` and `bundle deploy` could never converge past, reporting `1 to change` on every run ([#6060](#6060)). * Fixed a pipeline with `allow_duplicate_names: true` never converging on the direct engine: the field is only accepted on create/update and is never returned by the pipelines GET API, so every subsequent `bundle plan` reported the pipeline as a perpetual update. ([#6076](#6076)) * direct: A local change to an input-only field (one the API accepts on write but never returns on read, e.g. pipelines' `run_as` or external locations' `skip_validation`) is no longer silently skipped when the new value coincidentally matches the field's fabricated remote value. Previously such a change could hit the `remote_already_set` shortcut and be dropped from the plan. ([#6112](#6112)) * Revert usage of RedactiveSenstiveFields (added in [#5896](#5896), released in 1.10.0) which lead to incorrect behaviour (permanent drift) for duration field in Postgres resources ([#6179](#6179)). * Document postgres resource fields in the json schema ([#6164](#6164), [#6163](#6163)). * direct: Recreating a `vector_search_indexes` resource no longer fails with "Index ... is currently pending deletion" when the backend has not yet released the index name. The create is now retried until the name becomes available. ([#6143](#6143)) ### Dependency Updates * Bump `github.com/databricks/databricks-sdk-go` from v0.165.0 to v0.166.0. ([#6175](#6175)) * Upgrade Terraform provider to 1.124.0. ([#6174](#6174))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Let ai_runtime_task.code_source_path point at a local directory, not just a pre-built tarball. A new aicode mutator (run in the build phase, before libraries.ExpandGlobReferences/ReplaceWithRemotePath) detects a local-directory value, packages it into a reproducible, content-addressed tarball — honoring .gitignore and the top-level sync.include/exclude globs — uploads it to the user's ~/.air/repo_snapshots directory, and rewrites code_source_path to the uploaded (de-/Workspace-prefixed) path. Because it runs first, PR #5922's artifact-style collection then sees an already-remote path and skips it, so the two compose: a directory is packaged by the CLI, a pre-built .tgz still flows through the artifact path.
Also synthesizes a requirements.yaml next to the task's command_path from the job's serverless environments[] spec, so the AI Runtime harness sets up the workload environment. command_path translation itself is provided by #5922.
Verified end-to-end on a live workspace: a plain bundle deploy of a directory code_source_path yields a job whose run terminates SUCCESS; sync.exclude and .gitignore entries are absent from the uploaded snapshot; unchanged code skips re-upload.
Co-authored-by: Isaac
Changes
Adds an aicode bundle mutator that packages a local code directory referenced by an AI Runtime task's code_source_path and uploads it during bundle deploy, then wires it into the build/initialize phases.
Why
The SDK jobs.AiRuntimeTask.code_source_path field expects a workspace or UC-volume path to an uploaded code archive, and its doc comment states the CLI is responsible for packaging the user's local code into that archive. Nothing in DABs did that yet: a user pointing code_source_path at a local directory had no way to bundle deploy a runnable AI Runtime job.
This mutator implements that contract for bundles. It deliberately reuses DABs' existing local-artifact upload plumbing and runs before ReplaceWithRemotePath, so the two upload paths compose without special-casing: a directory is packaged and uploaded by this mutator (which then presents an already-remote path that the artifact collector skips), while a pre-built tarball delivered via an artifacts block still flows through the artifact path as a file. Content-addressing keeps redeploys of unchanged code a no-op. SynthesizeRequirements closes the loop so the deployed workload actually has its environment set up — without it the run fails during setup.
Tests
End-to-end validation
Verified on a live A10 staging workspace (dbc-04ac0685-8857) that the aicode mutator packages a local-directory code_source_path at bundle deploy and that the deployed AI Runtime workload runs to SUCCESS on real GPU. Covered all three code-source shapes the mutator must handle, plus the artifact-tarball path it must not claim.
What was validated
Seam: PackageAndUpload claims a local directory
Expected: dir → content-addressed .tar.gz, uploaded to ~/.air/repo_snapshots, code_source_path rewritten to
the remote archive
Result: ✅
────────────────────────────────────────
Seam: PackageAndUpload skips a local file
Expected: a pre-built tarball (artifacts block) flows through the standard artifact upload, not aicode
packaging
Result: ✅ (bundle/artifacts/ai_runtime_code_source acceptance test)
────────────────────────────────────────
Seam: command_path translation
Expected: rewritten to its absolute synced workspace path
Result: ✅
────────────────────────────────────────
Seam: SynthesizeRequirements
Expected: writes requirements.yaml next to command_path, derived from environments[].spec
Result: ✅
────────────────────────────────────────
Seam: GPU workload
Expected: extracts code, installs deps, runs on A10, exits SUCCESS
Result: ✅
Deployed ai_runtime_task after bundle deploy (mutator output)
{
"code_source_path": "/Users//.air/repo_snapshots/code_source/code_source_5659f5b55c3427b1.tar.gz",
"command_path": "/Workspace/Users//.bundle/convert-e2e-smoke/dev/files/command.sh"
}
code_source_path was ./code_source (a local directory) in the bundle — the mutator packaged it, uploaded the content-addressed archive, and rewrote the field to the remote path.
Reproduce