bundle: route config sync write-back to the block that defines each change - #6138
Merged
Conversation
This was referenced Aug 3, 2026
ilyakuz-db
commented
Aug 3, 2026
| + spark_version: [[DEFAULT_SPARK_VERSION]] | ||
| tasks: | ||
| - task_key: main | ||
| spark_version: [[DEFAULT_SPARK_VERSION]] |
Contributor
Author
There was a problem hiding this comment.
This is an expected change, the only changed field is still "spark_version" but new diff is less noisy,
ilyakuz-db
commented
Aug 3, 2026
| + node_type_id: [NODE_TYPE_ID] | ||
| + num_workers: 1 | ||
| + spark_version: 13.3.x-snapshot-scala2.12 | ||
| + - task_key: b_task_renamed |
Contributor
Author
There was a problem hiding this comment.
Same here, no changes in fields but diff is less noisy because task renames are now handled better
ilyakuz-db
force-pushed
the
configsync/split-list-write-back
branch
3 times, most recently
from
August 3, 2026 09:20
a1b1c01 to
fe07316
Compare
Collaborator
Integration test reportCommit: 9b75e0b
8 interesting tests: 4 RECOVERED, 4 SKIP
Top 6 slowest tests (at least 2 minutes):
|
ilyakuz-db
force-pushed
the
configsync/split-list-write-back
branch
4 times, most recently
from
August 3, 2026 12:15
d21a301 to
e81ef42
Compare
…hange Co-authored-by: Isaac
ilyakuz-db
force-pushed
the
configsync/split-list-write-back
branch
from
August 3, 2026 12:38
e81ef42 to
b09caf1
Compare
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
ilyakuz-db
force-pushed
the
configsync/split-list-write-back
branch
from
August 3, 2026 16:04
49300b3 to
8b3e0a8
Compare
Co-authored-by: Isaac
This was referenced Aug 3, 2026
…cross blocks Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
…nd FieldCandidates Co-authored-by: Isaac
ilyakuz-db
force-pushed
the
configsync/split-list-write-back
branch
4 times, most recently
from
August 4, 2026 09:42
5ad76e5 to
9cac42f
Compare
Co-authored-by: Isaac
ilyakuz-db
force-pushed
the
configsync/split-list-write-back
branch
from
August 4, 2026 09:51
9cac42f to
e803e40
Compare
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
denik
approved these changes
Aug 4, 2026
Co-authored-by: Isaac
Co-authored-by: Isaac
Collaborator
Integration test reportCommit: e7fb0c7
350 interesting tests: 311 MISS, 34 FAIL, 4 RECOVERED, 1 SKIP
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A resource's sequence field (e.g. job
tasks) can be written in two physical YAML regions: the top-levelresources.*block and atargets.override, either possibly in its own included file. Loading merges them into one list and sorts keyed lists by key.Config-sync resolves a change against that merged list, then writes it using the merged position, which causes out-of-bound indexing issues or issues with config corruption
Solution
bundle/configsync/blockindex.gomaps everydyn.Locationback to the block that owns it (not only a file), so a destination is looked up, andresolveSelectorsrewrites merged positions into block-local ones.Blocks are recovered by re-parsing the contributing files, because selecting a target destroys the distinction (it folds overrides into
resourcesand dropstargets). Re-parsing, rather than reloading via the mutator pipeline, avoids running the bundle'spreinitscript twice per sync.Guiding rule: correct write > no write > wrong write. Anything ambiguous is left for a later run.
How a destination is picked:
OverrideComputesettingexisting_cluster_id) counts as new: the value is present but carries no location, so no block declares it;Also worth knowing: index bookkeeping is per block, so a removal in one block cannot shift positions in another; rename pairing refuses ambiguous matches in both directions; and a target override may legitimately span several files, so a block is identified by (scope, file)
Tests
New acceptance tests, a lot of test cases to cover everything that this refactoring touched, but all are local as we only need to cover config resolution
Follow-ups
Stacked on this branch, each independently reviewable:
log.Debugfwhile the command prints the full changeset and exits 0, so a dropped edit is indistinguishable from an applied one.targets: {dev.eu: ...}). Pre-existing onmain, but it fails the whole command rather than skipping one change.Rebase onto
mainonce this merges.