feat: support SOURCE_DATE_EPOCH for reproducible OCI builds#753
Open
tsnaik wants to merge 4 commits intoproject-stacker:mainfrom
Open
feat: support SOURCE_DATE_EPOCH for reproducible OCI builds#753tsnaik wants to merge 4 commits intoproject-stacker:mainfrom
tsnaik wants to merge 4 commits intoproject-stacker:mainfrom
Conversation
tsnaik
requested review from
hallyn,
mikemccracken,
raharper,
rchamarthy,
rchincha,
smoser and
tych0
as code owners
Signed-off-by: Tanmay Naik <tnaik96@gmail.com>
Upgrade umoci from v0.5.0 to v0.6.0, which adds SOURCE_DATE_EPOCH support for clamping tar entry timestamps (similar to tar --clamp-mtime). When the SOURCE_DATE_EPOCH environment variable is set: - All tar layer timestamps are clamped to the epoch value via umoci's RepackOptions.SourceDateEpoch - OCI image config.created is set to the epoch timestamp - OCI history[].created entries use the epoch timestamp - The image author is stabilized to "stacker" instead of user@hostname - umoci.NewImage receives the epoch to clamp base image metadata This follows the reproducible-builds.org SOURCE_DATE_EPOCH specification, allowing bit-for-bit reproducible OCI image builds when the same epoch value and identical inputs are provided across builds. Also adds SOURCE_DATE_EPOCH to the default build environment passthrough list so the value is available inside build containers. Includes integration tests (test/reproducible.bats) covering: - Config timestamp and author field verification - Reproducibility across clean rebuilds (single and multi-layer) - Backward compatibility without SOURCE_DATE_EPOCH - Invalid input error handling - Different epoch values producing different timestamps Signed-off-by: Tanmay Naik <tnaik96@gmail.com>
tsnaik
force-pushed
the
add-reproducible-builds-support
branch
from
c5abc73 to
e9e6be6
Compare
Contributor
Author
|
something's up with the existing elasticsearch test case in |
tsnaik
force-pushed
the
add-reproducible-builds-support
branch
from
7230bcc to
38d1295
Compare
The build cache was not aware of SOURCE_DATE_EPOCH. When a layer was cached from a build with one epoch value, subsequent builds with a different epoch would hit the cache and silently reuse the old timestamps and author metadata. Add SourceDateEpoch to CacheEntry and compare it during Lookup(). A change in the epoch now triggers a cache miss and full rebuild. Signed-off-by: Tanmay Naik <tnaik96@gmail.com>
Upstream commit[1] removed the Dockerfile from the main branch, causing this test to fail. ``` dockerfiles/elasticsearch: No such file or directory ``` Clone an older tag that still contains the Dockerfile. [1]: elastic/dockerfiles@a25b223 Signed-off-by: Tanmay Naik <tnaik96@gmail.com>
tsnaik
force-pushed
the
add-reproducible-builds-support
branch
from
38d1295 to
29ce1d6
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #753 +/- ##
==========================================
+ Coverage 52.84% 52.93% +0.08%
==========================================
Files 59 59
Lines 6492 6535 +43
==========================================
+ Hits 3431 3459 +28
- Misses 2418 2431 +13
- Partials 643 645 +2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
|
all but the arm64 unpriv tests passed. That one printed the 26th test output and then looks like maybe it timed out? then nothing. the unpriv amd64 tests have this test sequence: test 28 took almost an hour. Maybe the skipped test 27 didn't make it to the output and the long test 28 caused a timeout? Do we really need a test that takes 48 minutes on its own? maybe we should mark it as broken |
Contributor
Author
|
passed on re-trigger. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
feature
Which issue does this PR fix:
N/A
What does this PR do / Why do we need it:
Adds support for the SOURCE_DATE_EPOCH environment variable to enable bit-for-bit reproducible OCI image builds. Upgrades umoci from v0.5.0 to v0.6.0, which provides timestamp clamping in tar layers via
RepackOptions.SourceDateEpoch.When
SOURCE_DATE_EPOCHis set:tar --clamp-mtime)config.createdis set to the epoch timestamphistory[].createdentries use the epoch timestamp"stacker"instead ofuser@hostnameumoci.NewImagereceives the epoch to clamp base image metadataSOURCE_DATE_EPOCHis also added to the default build environment passthrough list so the value is available insiderun:sections.If an issue # is not available please add repro steps and logs showing the issue:
Without
SOURCE_DATE_EPOCH, building the same stacker YAML twice produces images with different digests because timestamps and author fields vary between runs. With this change:Testing done on this change:
Integration tests added in
test/reproducible.batscovering 9 scenarios. Manual verification confirmed identical manifest, config, and layer digests across clean rebuilds with the same epoch.Manual test
Automation added to e2e:
Yes —
test/reproducible.batswith 9 test cases:SOURCE_DATE_EPOCHsetsconfig.createdtimestamp correctly"stacker"when epoch is setSOURCE_DATE_EPOCH, author remainsuser@hostname(backward compat)SOURCE_DATE_EPOCHproduces an errorSOURCE_DATE_EPOCHis available insiderun:sectionsreproducible.bats output
Will this break upgrades or downgrades?
No. Behavior is unchanged when
SOURCE_DATE_EPOCHis not set.Does this PR introduce any user-facing change?:
Yes — users can now set
SOURCE_DATE_EPOCHto produce reproducible OCI images.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.