Skip to content

Conversation

@leodido
Copy link
Contributor

@leodido leodido commented Nov 20, 2025

Problem

When exportToCache: true is enabled, Docker images are exported as OCI layout and not loaded into the Docker daemon. This causes docker inspect to fail when generating SLSA provenance subjects.

Error in CI:

package build failed while building
Reason: failed to inspect image e25414d36aab81809a0d8c723ea5827badbb3baf: exit status 1
Error: No such object: e25414d36aab81809a0d8c723ea5827badbb3baf

Root cause:

  1. OCI layout export: docker buildx build --output type=oci,dest=image.tar
  2. Image exported to OCI layout (not in Docker daemon) ✅
  3. createDockerSubjectsFunction calls docker inspect <version>
  4. Fails because image doesn't exist in daemon ❌

Why Wasn't This Caught?

The integration test TestDockerPackage_OCILayout_Determinism_Integration doesn't enable SLSA, so the Subjects function is never called during testing. The bug only manifests when:

  • exportToCache: true (OCI layout)
  • SLSA provenance generation is enabled
  • Both conditions together

Solution

Extract the image digest from OCI layout files instead of using docker inspect.

Design Decision: Separate Functions

Instead of one function with conditional logic, we now have two separate functions:

  1. createDockerInspectSubjectsFunction() - Legacy push workflow

    • Image is in Docker daemon
    • Uses docker inspect to get digest
    • Called when exportToCache: false
  2. createOCILayoutSubjectsFunction() - OCI layout export

    • Image is NOT in Docker daemon
    • Extracts digest from OCI layout files
    • Called when exportToCache: true

Why separate functions?

  • ✅ Single responsibility - each function has one purpose
  • ✅ No conditional logic - simpler to understand
  • ✅ Easier to test - each path is independent
  • ✅ Better maintainability - changes to one don't affect the other

Changes

  1. New function: extractDigestFromOCILayout()

    • Reads index.json from OCI layout directory
    • Parses manifest digest from index
    • Returns digest in SLSA format
  2. New function: createDockerInspectSubjectsFunction()

    • Replaces old function for legacy path
    • Uses docker inspect (unchanged behavior)
  3. New function: createOCILayoutSubjectsFunction()

    • New function for OCI layout path
    • Extracts image.tar to temporary directory
    • Calls extractDigestFromOCILayout() to get digest
  4. Updated call sites:

    • Legacy path: createDockerInspectSubjectsFunction(version, cfg)
    • OCI layout path: Set up in PostProcess with buildDir available

OCI Layout Structure

image.tar/
├── oci-layout          # Format version
├── index.json          # Image index with manifest digest
└── blobs/
    └── sha256/
        ├── abc123...   # Layer blobs
        └── def456...   # Config blob

Digest extraction:

// index.json
{
  "manifests": [
    {
      "digest": "sha256:abc123...",  // ← Extract this
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "size": 1234
    }
  ]
}

Testing

Unit Tests (10 tests)

New file: pkg/leeway/build_oci_test.go

1. extractDigestFromOCILayout() - 6 tests

  • ✅ Valid OCI index with single manifest
  • ✅ Valid OCI index with multiple manifests (uses first)
  • ✅ Empty manifests array (error)
  • ✅ Invalid digest format (error)
  • ✅ Invalid JSON (error)
  • ✅ Missing index.json file (error)

2. createOCILayoutSubjectsFunction() - 2 tests

  • ✅ Full workflow with OCI layout
  • ✅ Missing image.tar (error)

3. createDockerInspectSubjectsFunction() - 1 test

  • ✅ Function structure verification

4. Behavior documentation - 1 test

  • ✅ Documents that functions work with/without SLSA
$ go test -v ./pkg/leeway/ -run "TestExtract|TestCreate|TestSubjects"
=== RUN   TestExtractDigestFromOCILayout
--- PASS: TestExtractDigestFromOCILayout (0.00s)
=== RUN   TestCreateOCILayoutSubjectsFunction
--- PASS: TestCreateOCILayoutSubjectsFunction (0.00s)
=== RUN   TestCreateDockerInspectSubjectsFunction
--- PASS: TestCreateDockerInspectSubjectsFunction (0.00s)
=== RUN   TestSubjectsFunctionBehavior
    build_oci_test.go:320: ✅ Both functions can be set up regardless of SLSA state
    build_oci_test.go:321: ✅ Functions are only called when SLSA provenance generation is active
--- PASS: TestSubjectsFunctionBehavior (0.00s)
PASS
ok      github.com/gitpod-io/leeway/pkg/leeway  0.031s

Integration Test (1 test)

New test: TestDockerPackage_OCILayout_SLSA_Integration

Tests the full workflow:

  • ✅ Creates docker-container builder (required for OCI export)
  • ✅ Builds package with exportToCache: true and SLSA enabled
  • ✅ Verifies build succeeds without docker inspect error
  • ✅ Confirms OCI layout created (image.tar in cache)
$ go test -v -tags=integration -run TestDockerPackage_OCILayout_SLSA_Integration ./pkg/leeway/
=== RUN   TestDockerPackage_OCILayout_SLSA_Integration
    build_integration_test.go:968: ✅ Build succeeded with OCI layout export
    build_integration_test.go:969: ✅ No 'docker inspect' error occurred
    build_integration_test.go:970: ✅ This confirms the fix works
--- PASS: TestDockerPackage_OCILayout_SLSA_Integration (4.23s)
PASS
ok      github.com/gitpod-io/leeway/pkg/leeway  4.254s

Test Coverage Summary

Both code paths tested:

  • Legacy path (docker inspect) - existing + new tests
  • OCI layout path (OCI extraction) - new tests

Both SLSA scenarios tested:

  • With SLSA enabled - integration test
  • Without SLSA enabled - documented behavior

All edge cases covered:

  • Valid inputs
  • Invalid inputs
  • Missing files
  • Error handling

Backward Compatibility

  • ✅ Legacy path (docker inspect) unchanged
  • ✅ Only affects OCI layout + SLSA combination
  • ✅ No breaking changes
  • ✅ All existing tests pass

Impact

Enables:

  • ✅ SLSA L3 provenance with OCI layout
  • ✅ Deterministic Docker image caching
  • ✅ Full SLSA compliance in the monorepo CI

Related

@leodido leodido self-assigned this Nov 20, 2025
@leodido leodido force-pushed the ldd/fix-oci-slsa-inspect branch from e7d488c to 96cc3dd Compare November 20, 2025 19:01
When exportToCache is enabled, Docker images are exported as OCI layout
and not loaded into the Docker daemon. This causes 'docker inspect' to
fail when generating SLSA provenance subjects.

Problem:
- OCI layout export creates image.tar without loading to Docker daemon
- createDockerSubjectsFunction calls 'docker inspect <version>'
- Fails with 'No such object: <image-id>'
- Blocks SLSA provenance generation

Root Cause:
- Integration test for OCI layout doesn't enable SLSA
- Subjects function never called during testing
- Bug went undetected

Solution:
- Split into two separate functions for clarity:
  * createDockerInspectSubjectsFunction() - for legacy push workflow
  * createOCILayoutSubjectsFunction() - for OCI layout export
- Add extractDigestFromOCILayout() to read digest from OCI index.json
- Extract image.tar and parse index.json for manifest digest
- Use digest for SLSA provenance subjects

Changes:
- Separate functions for Docker inspect vs OCI layout paths
- Extract image.tar to temporary directory
- Read index.json from OCI layout
- Parse manifest digest from index
- Use digest for SLSA provenance subjects

Testing:
- Unit tests for extractDigestFromOCILayout() (6 test cases)
- Integration test for OCI layout + SLSA (TestDockerPackage_OCILayout_SLSA_Integration)
- All tests pass
- Maintains backward compatibility

Fixes: gitpod-io/gitpod-next#11869

Co-authored-by: Ona <[email protected]>
Copy link
Member

@geropl geropl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM

@leodido leodido merged commit 61f08fe into main Nov 21, 2025
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants