Skip to content

Write config files atomically to prevent volume-keys.toml corruption - #2696

Open
changren-wcr wants to merge 1 commit into
buildpacks:mainfrom
changren-wcr:fix/atomic-config-write
Open

changren-wcr wants to merge 1 commit into
buildpacks:mainfrom
changren-wcr:fix/atomic-config-write

Conversation

@changren-wcr

@changren-wcr changren-wcr commented Sep 18, 2026 •

Copy link
Copy Markdown

Summary

config.Write truncated the target file and streamed the TOML encoding into it in place (os.Create + toml.NewEncoder(w).Encode). When several pack processes write the same file at the same time, as parallel pack build runs do with volume-keys.toml, readers can see a partially written file and writers can interleave. This can leave a corrupted file behind, after which every later pack build on the host fails (#2695).

This PR changes config.Write to encode into a temporary file in the same directory (os.CreateTemp) and then os.Rename it over the target. Readers now always see either the old or the new complete file.

Behavior details:

  • Encoding errors no longer truncate the existing file; it is left untouched.
  • File mode: an existing file keeps its mode. A newly created file is 0600, where os.Create previously gave 0666 minus the umask. The umask can't be read portably, and 0600 is the safer default for files under PACK_HOME.
  • Read-only files: an existing file the user made read-only still makes Write fail, instead of being replaced by the rename (which only needs write access to the directory). This keeps the existing returns clear error if fails to write command tests passing unchanged.

Out of scope, possible follow-up: getVolumeKey still does an unlocked read-modify-write, so concurrent builds can drop each other's newly generated keys. A dropped key only means a key is generated again later (a cache miss), not a failure. Fixing it needs a cross-platform file lock, which I left out of this PR to keep it small.

Output

Reproduction script from #2695 (empty app, unique tag per build, PACK_HOME isolated, file seeded with 8000 entries). Both runs were 16 builds in parallel on the same Linux host.

Before

Official v0.40.9: the file was permanently corrupted after the first round.

ERROR: failed to build: executing lifecycle: failed to read config file at path /tmp/pack-race-repro/home/volume-keys.toml: toml: line 8004 (last key "volume-keys"): strings cannot contain newlines

(A separate run with 8 in parallel had 78/160 builds fail with files cannot contain NULL bytes / unexpected EOF.)

After

This branch, 10 rounds × 16 builds: 0/160 builds failed on volume-keys.toml. All 160 went on to detect (which fails as expected for the empty app). The file stayed valid throughout and no temporary files were left behind.

Tests

New cases in internal/config/config_test.go:

  • Concurrent writes and reads (8 writers + 8 readers on one file): no read errors, and the final file is valid. It fails on main with the same NULL bytes / unexpected EOF errors seen in production. It is skipped on Windows, where renaming over a file another goroutine has open can fail with "Access is denied". I have no Windows machine to verify that there.
  • Encoding error: the existing file is left untouched. On main it is truncated.
  • Successful writes: no temp files are left behind, and an existing file's mode is preserved.

Checks I ran locally:

  • Passing: go test ./internal/config/ ./internal/commands/, -race -count=10 on internal/config, and golangci-lint (0 issues).
  • GOOS=windows go vet and a test compile both succeed.
  • pkg/cache: the only failures are the Docker-daemon tests, which fail the same way on main on my machine (no Docker socket).

Documentation

  • Should this change be documented?
    • Yes, see #___
    • No

Related

Resolves #2695

config.Write truncated the target file and streamed the TOML encoding into
it in place. When several pack processes write the same file concurrently,
as parallel `pack build` runs do with volume-keys.toml, readers can observe
a partially written file and writers can interleave, leaving a corrupted
file behind that makes every subsequent `pack build` on the host fail with
"failed to read config file at path .../volume-keys.toml".

Encode into a temporary file in the same directory and rename it over the
target instead, so readers always see either the old or the new complete
file. An encoding error now also leaves the existing file untouched.

The mode of an existing file is preserved, and an existing read-only file
still makes Write fail instead of being silently replaced.

Fixes buildpacks#2695

Signed-off-by: changren-wcr <105254603+changren-wcr@users.noreply.github.com>
@changren-wcr
changren-wcr requested review from a team as code owners September 18, 2026 07:03
@github-actions github-actions Bot added the type/enhancement Issue that requests a new feature or improvement. label Sep 18, 2026
@github-actions github-actions Bot added this to the 0.41.0 milestone Sep 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/enhancement Issue that requests a new feature or improvement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent pack build corrupts $PACK_HOME/volume-keys.toml (non-atomic, unlocked read-modify-write)

1 participant