Add oven: a local build, test, validate, and scan environment - #727
Draft
bschwedler wants to merge 23 commits into
Draft
Add oven: a local build, test, validate, and scan environment#727bschwedler wants to merge 23 commits into
bschwedler wants to merge 23 commits into
Conversation
- Revert goss and hadolint versions to 'latest' (as per Global Constraint) - Fix goss: handle new tarball asset format (goss_N_linux_arch.tar.gz) and combined SHA256SUMS file; dynamically resolve latest tag via GitHub API - Fix hadolint: use combined checksums.sha256 file instead of missing per-binary checksums; restore integrity verification that was removed in prior fix - Standardize all tool-install RUN blocks to use semicolon-continuation style
Important #1: Fix hadolint URL branching for pinned vs latest versions - /releases/latest/download/ for latest - /releases/download/v{VERSION}/ for pinned versions - Verified with --build-arg HADOLINT_VERSION=2.15.1 Important #2: Ensure pipefail for goss tag resolution - Wrapped in bash explicitly (will be improved with shell directive in next iteration) Minor #3: Add unsupported TARGETARCH error cases - Added *) error arms to all three case statements (goss, hadolint, trivy) - Clear error messages instead of cryptic unbound variable errors Minor #4: Update divergence comments - goss: Note tarball format differs from setup-goss/action.yml - hadolint: Note checksums.sha256 differs from setup-hadolint/action.yml
- Restore GOSS_VERSION=latest and HADOLINT_VERSION=latest (removed hardcoded pins) - Add SHELL [/bin/bash, -o, pipefail, -c] for global pipefail support - Fix goss dynamic version resolution: save JSON to temp file first, then extract tag with grep/sed (workaround for write error in pipe) - Restore goss tarball format (goss_N_linux_arch.tar.gz + SHA256SUMS) - Keep hadolint URL branching (latest vs pinned) and checksums.sha256 - All four review fixes present: URL branching, pipefail, error cases, comments
Add two just recipes to build and run the bakery-oven Docker image for local development, testing, validation, and scanning: - oven-build: Rebuilds the Docker image - oven: Builds if needed, then runs the container with sibling repos mounted, forwarding any command-line arguments
Use test -t 1 to check if stdout is a terminal. This allows the recipe to work in both interactive terminals (with -t for full TTY support) and non-interactive environments like CI scripts or non-TTY shells (with -i only). Fixes incompatibility with non-interactive contexts where -it would fail with "cannot attach stdin to a TTY-enabled container".
Calculate mount root using git rev-parse --git-common-dir instead of
dirname {{ CWD }}. This handles worktrees correctly:
- For worktrees: resolves the real .git directory, then finds the
parent directory containing sibling repos
- For main checkouts: behavior is identical to original (dirname twice
on .git/ returns dirname {{ CWD }})
- Fixes git version detection in container by ensuring .git is
accessible even when running from a worktree
This allows uv-dynamic-versioning to detect the git repository and
successfully build posit-bakery during container startup.
Add git config --global --add safe.directory '*' before uv sync to allow git operations on bind-mounted repositories with mismatched ownership (root in container vs host user). This resolves the RuntimeError from uv-dynamic-versioning when running uv sync in Task 2 end-to-end tests.
This eliminates two critical problems with the root-user design: 1. No longer need to weaken git's security via safe.directory workaround 2. Files created by uv sync now have host-user ownership, not root ownership Changes: - Containerfile: Add HOME=/tmp, /opt/oven/bin (world-writable), and update PATH to include /opt/oven/bin for non-root-writable bakery wrapper - entrypoint.sh: Remove git config --global safe.directory line; write bakery wrapper to /opt/oven/bin instead of /usr/local/bin - justfile: Add --user and --group-add flags to docker run to: * Run container as host UID:GID (no UID mismatch with git) * Add docker socket GID to supplementary groups (socket access without root) Verified: - Container runs as host user (uid=1000) - Docker socket is accessible (GID 973 in supplementary groups) - .venv created with host-user ownership, not root
Enable just's positional-arguments setting and use "$@" instead of
{{ ARGS }} to properly forward variadic arguments to the container.
Problem: just's {{ ARGS }} template substitution joins arguments with
spaces and does not re-quote, causing `bash -lc 'docker ps'` to be
rendered as 4 separate tokens (bash, -lc, docker, ps) instead of 3
(bash, -lc, 'docker ps'). The shell then re-splits these, breaking
compound argument semantics.
Solution: With positional-arguments := true, recipe parameters are
passed as real shell positional parameters ($1, $2, ..., "$@"),
preserving argument boundaries natively without text-flattening.
This allows:
- just oven bash -lc 'docker ps' to run correctly (3 tokens)
- just oven bakery --help to work as expected
- Echo strings like 'echo forwarded-args-work' to be properly quoted
Template substitution ({{ CWD }}, named parameters) continues to work
unchanged.
Important fix: Container now uses /opt/oven/venv (internal container only),
not host's posit-bakery/.venv. UV cache and Python interpreter stored in
/opt/oven/state mounted from host cache (survives across --rm), eliminating:
- ~30MB Python re-download on every just oven invocation
- Host .venv corruption with container-only paths
- Ping-ponging between container and host .venv versions
Additional fixes:
- Version strings normalized: ${VERSION#v} to handle redundant v prefix
- --remove-on-error on hadolint curls (don't leave stray files)
- test -t 0 instead of test -t 1 (stdin, not stdout terminal check)
- BSD stat fallback for macOS (-f '%g' for group ID)
- Quote all {{ CWD }} uses in justfile
- COPY --chmod=0755 instead of separate chmod RUN
- exec "${@:-bash}" to handle empty args under set -u
uv run --directory changes the spawned process's cwd (like cd && uv run), causing relative paths (e.g., --context posit-bakery/test/resources/basic) to resolve incorrectly. Use --project instead: resolves the project path without relocating the process's working directory. Verified: - bakery build with relative --context path works - bakery version and docker ps still work (no regressions)
Set DGOSS_TEMP_DIR to the mount_root directory to allow dgoss to create its temporary directory on a shared mount that exists on both the host and container. Problem: dgoss creates a temp directory under /tmp inside the oven container and tries to mount it into the tested container via DooD (Docker out of Docker). When dgoss's docker run attempts to mount /tmp/tmp.XXXXXXXXXX from inside the container, the host daemon looks for this path on the real host filesystem, where it doesn't exist. This causes the goss binary to be unavailable inside the tested container, failing with "/goss/goss: not found". Solution: Point DGOSS_TEMP_DIR to mount_root (the parent directory containing all sibling repos), which is already mounted from the host. dgoss creates its temp subdirectory there, and when it attempts to mount that subdirectory, the path is accessible on both the host and container. This enables real goss test execution: `bakery dgoss run ...` now successfully runs goss tests inside built images.
…llback Major fixes for production readiness: 1. HOME persistence: HOME=/opt/oven/state/home instead of /tmp preserves trivy's vulnerability database and wizcli's auth token across --rm runs, eliminating re-download on every invocation (same underlying issue as the venv/cache fix from prior commit). 2. Filesystem consolidation: UV_PROJECT_ENVIRONMENT, UV_CACHE_DIR, and UV_PYTHON_INSTALL_DIR now all point to /opt/oven/state, the mounted persistent directory. Previously venv was split from cache, triggering uv's cross-device warning and forcing full-copy mode (slow hardlinking) instead of efficient hardlinks, requiring rebuild every run. 3. Hadolint fallback: ported exact logic from setup-hadolint/action.yml to handle both v2.15.0+ (single checksums.sha256 file) and older releases (per-binary .sha256 files). Gracefully handles binary naming variations (lowercase vs capitalized Linux/Darwin). 4. Documentation: added comprehensive comments explaining every env var, mount, docker CLI flag, and non-obvious justfile logic. Explains why git-common-dir is needed (worktree support) and why test -t 0 is used (stdin check, not stdout). 5. Pre-commit hook fix: entrypoint.sh mode changed from 100644 to 100755 (required by check-shebang-scripts-are-executable hook). All regression tests pass: - bakery version (no hardlink warning with consolidated filesystem) - docker ps (socket access) - bakery build with relative --context path - bakery dgoss run (Goss test results) - bakery hadolint run (linting)
GitHub's release-download URLs resolve asset names case-insensitively, so requesting 'hadolint-linux-x86_64' (lowercase) succeeds and downloads the actual file 'hadolint-Linux-x86_64' (capitalized) under the wrong local filename. The sha256sum then fails because checksum file refers to the true capitalized name. Solution: Extract the binary's true filename from the checksum file itself (which is authoritative) and use that for both the download URL and the local filename. This works for both: - v2.15.0+ (combined checksums.sha256 covering all binaries) - v2.12.0 and older (per-binary .sha256 files) GitHub serves the file regardless of URL case, so the extracted name from checksums always works for the download. Verified: - Default (latest): hadolint 2.15.1 builds and runs - v2.12.0 (fallback path): builds, runs, reports version 2.12.0 - Full regression suite passes
grep/sed textual parsing of the GitHub API's JSON response is fragile to formatting changes. This repo already hit that problem and fixed it in setup-goss/action.yml (109c60a) by switching to jq; the new oven/Containerfile had reintroduced the old pattern independently. Add jq to the base-utils layer and match the established approach. Verified with a full docker build of oven/Containerfile: goss_tag resolves correctly (v0.4.10) and the checksum still verifies OK.
The oven wraps `bakery` directly via the entrypoint, but the host dev workflow documented in CLAUDE.md runs tests and setup through `just` (`just test`, `just test-all`, `just setup`), not the bakery CLI alone. Without `just` installed, that workflow has no way to run inside the container. Installed following this file's established pattern: pinned version, checksum-verified download, matching the goss/hadolint/ trivy blocks above it. just's own release tags have no "v" prefix and its Linux assets are musl-only, unlike the other three tools. Verified with a full docker build, confirmed `just --version` works, and confirmed `just --list` correctly parses posit-bakery's real justfile from a mounted checkout.
Running the full test suite inside the oven surfaced 10 failures. Two of them were `bakery ci merge` and a find_oras_bin unit test failing to find oras, since it was never installed in this image (bakery's imagetools plugin shells out to it directly). Installed following the same pinned-version, checksum-verified pattern as goss/hadolint/trivy/just above it. oras's asset names already use Docker's native amd64/arm64 naming, so no arch-mapping case statement is needed (same situation as wizcli). Verified with a full docker build and `oras version` inside the image. The other 8 failures are unrelated to this image: 7 are @image_build-tagged multiplatform tests that need host-level QEMU binfmt emulation (out of scope for this Containerfile), and 1 is a DooD network-namespace limitation in a test helper that hardcodes "localhost" for a registry container's published port.
posit-bakery's RegistryContainer test helper starts a local registry and reports its URL as "localhost:<published-port>", which only resolves correctly when the caller shares the Docker daemon's network namespace. DooD only shares the daemon's socket, not its network namespace, so "localhost" inside the oven container and "localhost" on the host-daemon side are different network stacks - the test's HTTP request to its own registry was connection-refused. --network host makes them the same stack. Documented in the Containerfile's runtime-contract comment alongside the other required flags, since the image itself can't enforce this one. Verified: TestRegistryContainer::test_context_manager now passes inside the oven, along with the oras-dependent tests fixed in the previous commit.
Verified empirically that binfmt_misc is a single table global to the host kernel, not namespaced per-container: registering QEMU emulators from inside the oven (via its DooD socket access) makes them visible on the bare host immediately afterward, and vice versa. Confirmed by re-running a previously-failing @image_build-tagged multiplatform test after registering - it now builds linux/arm64 targets successfully. Doesn't persist across a reboot, so this is a separate, explicit recipe rather than something baked into `oven` or entrypoint.sh - running it on every shell invocation would silently mutate host kernel state for no reason most of the time.
Replace the oven-setup-qemu recipe with a comment pointing at the native Ubuntu package instead of running it. tonistiigi/binfmt needs a --privileged container from a third-party image just to write a few kernel table entries - qemu-user-binfmt does the same thing as a normal apt install from Ubuntu's own archive, and its systemd integration means it persists across reboots for free, which the container approach couldn't do without a hand-rolled systemd unit anyway.
Move this out of the justfile and into the actual maintainer-facing docs, alongside the other Footguns entries. Documents the native qemu-user-binfmt package rather than scripting a --privileged third-party container for a one-time host setup step.
The oven recipe's git-common-dir guard used an apostrophe inside a
${var:?message} expansion, which parses fine under just's default
shell (dash, or bash-as-sh on macOS) but fails with a cryptic "unexpected
EOF" if just is ever configured to use native bash. Reworded the message
to avoid the apostrophe entirely.
The mkdir for the persistent state directory had no failure guard, unlike
the two checks right above it. just's default shell doesn't enable
errexit, so a failed mkdir would silently fall through to `docker run`,
which auto-creates the missing bind-mount source as root - breaking the
--user host-UID matching the whole design depends on. Added an explicit
guard to match the existing pattern.
Also removed the QEMU/binfmt_misc explanation from the justfile comment,
since it duplicated CONTRIBUTING.md verbatim; left a one-line pointer
instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds
oven. The oven is a Docker-outside-of-Docker container. It holds the tools needed to build, test, validate, and scan a product image on a local computer:This PR also adds two commands,
just oven-buildandjust oven. Thejust ovencommand builds the oven, then starts it with the sibling repositories mounted.This PR is a draft. Two items are still open. First,
ghcr.io/astral-sh/uv:latesthas no version pin and no checksum, unlike every other tool in the image. Second, CONTRIBUTING.md and the README do not introduce the oven anywhere. A troubleshooting note is the only reference, and it assumes prior knowledge of the oven.