Write config files atomically to prevent volume-keys.toml corruption - #2696
Open
changren-wcr wants to merge 1 commit into
Open
changren-wcr wants to merge 1 commit into
changren-wcr wants to merge 1 commit into
Conversation
config.Write truncated the target file and streamed the TOML encoding into it in place. When several pack processes write the same file concurrently, as parallel `pack build` runs do with volume-keys.toml, readers can observe a partially written file and writers can interleave, leaving a corrupted file behind that makes every subsequent `pack build` on the host fail with "failed to read config file at path .../volume-keys.toml". Encode into a temporary file in the same directory and rename it over the target instead, so readers always see either the old or the new complete file. An encoding error now also leaves the existing file untouched. The mode of an existing file is preserved, and an existing read-only file still makes Write fail instead of being silently replaced. Fixes buildpacks#2695 Signed-off-by: changren-wcr <105254603+changren-wcr@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
config.Writetruncated the target file and streamed the TOML encoding into it in place (os.Create+toml.NewEncoder(w).Encode). When several pack processes write the same file at the same time, as parallelpack buildruns do withvolume-keys.toml, readers can see a partially written file and writers can interleave. This can leave a corrupted file behind, after which every laterpack buildon the host fails (#2695).This PR changes
config.Writeto encode into a temporary file in the same directory (os.CreateTemp) and thenos.Renameit over the target. Readers now always see either the old or the new complete file.Behavior details:
0600, whereos.Createpreviously gave0666minus the umask. The umask can't be read portably, and0600is the safer default for files underPACK_HOME.Writefail, instead of being replaced by the rename (which only needs write access to the directory). This keeps the existingreturns clear error if fails to writecommand tests passing unchanged.Out of scope, possible follow-up:
getVolumeKeystill does an unlocked read-modify-write, so concurrent builds can drop each other's newly generated keys. A dropped key only means a key is generated again later (a cache miss), not a failure. Fixing it needs a cross-platform file lock, which I left out of this PR to keep it small.Output
Reproduction script from #2695 (empty app, unique tag per build,
PACK_HOMEisolated, file seeded with 8000 entries). Both runs were 16 builds in parallel on the same Linux host.Before
Official
v0.40.9: the file was permanently corrupted after the first round.(A separate run with 8 in parallel had 78/160 builds fail with
files cannot contain NULL bytes/unexpected EOF.)After
This branch, 10 rounds × 16 builds: 0/160 builds failed on
volume-keys.toml. All 160 went on todetect(which fails as expected for the empty app). The file stayed valid throughout and no temporary files were left behind.Tests
New cases in
internal/config/config_test.go:mainwith the sameNULL bytes/unexpected EOFerrors seen in production. It is skipped on Windows, where renaming over a file another goroutine has open can fail with "Access is denied". I have no Windows machine to verify that there.mainit is truncated.Checks I ran locally:
go test ./internal/config/ ./internal/commands/,-race -count=10oninternal/config, andgolangci-lint(0 issues).GOOS=windows go vetand a test compile both succeed.pkg/cache: the only failures are the Docker-daemon tests, which fail the same way onmainon my machine (no Docker socket).Documentation
Related
Resolves #2695