test(podman): run existing Podman behavioral e2e coverage in CI #3712

Activity

1. johntmyers

   added

   topic:testing

   area:compute

   on Sep 25, 2026
2. politerealism commented on Sep 27, 2026

   

   politerealism

   on Sep 27, 2026

   Contributor

   More actions

   Looking at this one
3. politerealism commented on Sep 27, 2026

   

   politerealism

   on Sep 27, 2026

   Contributor

   More actions

   Plan and order of operations

   This issue overlaps significantly with open PR #3637 ("test(e2e): run podman suite with tmachine", author elezar), which independently adds a tmachine-based e2e-podman nextest archive (43 tests passing) covering most of the 39 eligible e2e-podman targets, removes podman_userns.rs as superseded by the driver-podman tmachine suite (resolving that acceptance criterion directly), and depends on PR #3597 (migrates sync.rs coverage into conformance scenarios). Both #3637 and #3597 are currently green in CI but show mergeable: CONFLICTING and need a rebase before they can land.

   Order of operations for this work:

   1. Treat test(e2e): run podman suite with tmachine #3637 + test(conformance): migrate file transfer coverage #3597 as prerequisites. This plan does not duplicate their work and does not touch the files they modify (tests/ansible/playbooks/drivers/podman/e2e.yaml, tests/artifacts.nix) until they're on main.
   2. Track A (starting now, independent of test(e2e): run podman suite with tmachine #3637):
      • Wire podman_preflight into CI via the existing podman-external-driver-e2e job's already-exported driver binary artifact (it never runs anywhere today because that job's mise task disables the cargo-test step for other reasons).
      • Add an automated inventory/reachability check (new tasks/scripts/check-podman-e2e-coverage.sh, modeled on check-cargo-lockfiles.sh) that fails CI if a new e2e-podman-eligible test target isn't accounted for in either PODMAN_CI_TESTS or a documented exclusion — this directly satisfies the acceptance criterion preventing silent future omissions like this one.
      • Document the current test-selection contract in CI.md.
   3. Track B (after test(e2e): run podman suite with tmachine #3637 + test(conformance): migrate file transfer coverage #3597 merge):
      • Extend the coverage check to also read test(e2e): run podman suite with tmachine #3637's podmanE2eFollowUpBinaries exclusion list; file tracked follow-up issues for the exclusions that need distinct scope work (sandbox_lifecycle conformance migration, podman_oci_identity SPIFFE fixture plumbing, provider_refresh_handles's own purpose-built suite), and link transparent_tcp's exclusion to the existing test(e2e): restore musl DNS probe coverage for guest prebuilt artifacts #3009 rather than duplicating it.
      • Investigate the remaining plausible quick-fix targets (provider_auto_create, proxy_egress_pipeline, websocket_conformance, workspace_lifecycle) against a live tmachine environment.
      • Add a rootful leg to the new tmachine e2e-podman testsuite (its current playbook hardcodes rootless-only values instead of using the tmachine_container_runtime role the way driver-podman's playbooks already do).
      • Wire driver-podman/e2e-podman into Release Dev and Release Tag qualification, mirroring the existing provider-refresh job pattern.
      • Add visible selected-vs-eligible test-count reporting so a narrow run can't be mistaken for full coverage.

   PRs for Track A will follow this comment. Track B PRs will follow once #3637/#3597 land.
4. politerealism commented on Sep 29, 2026

   

   politerealism

   on Sep 29, 2026

   Contributor

   More actions

   Status update given everything that's landed since the plan above:

   • podman_preflight is resolved via test(podman): move podman_preflight into driver-podman integration tests #3783 (reopened and approved by @elezar, /ok-to-test applied, pending merge) rather than the original test(podman): wire podman_preflight into CI and add a Podman e2e coverage check #3749 approach.
   • podman_userns is resolved via test(e2e): run podman suite with tmachine #3637 (superseded by the driver-podman tmachine suite).
   • podman_resource_limits now runs for real against a live gateway via test(e2e): run podman suite with tmachine #3637's tmachine e2e-podman archive.

   @elezar — given #3460 (Nix/tmachine target-state doc) and the string of test(conformance)/test(tmachine) PRs you've got in flight (#3766, #3768, #3792, #3795, #3839, etc.), it looks like most of the remaining shared-target and driver-specific coverage gaps here (the 6 of 12 shared targets still excluded, podman_oci_identity, provider_refresh_handles) will fall out of that migration rather than needing separate Podman-specific CI wiring. To avoid duplicating effort, I'm deliberately not touching that migration.

   What I'll pick up next instead, since it's orthogonal to the conformance/driver-specific split:

   • Release Dev and Release Tag bundled Podman behavioral qualification (currently absent from both).
   • Selected-vs-eligible test-count reporting in job names/summaries, so a narrow run can't be mistaken for full coverage.
   • Documenting the rootless-only scope of the current e2e-podman tmachine archive (only fedora-podman-rootless runs it; podman_resource_limits specifically would benefit from a rootful leg since it reads real cgroup v2 state and rootless cgroup delegation is the harder/riskier case).

   Flag anything above that's already in flight or planned elsewhere so we don't collide again.
5. elezar commented on Sep 30, 2026

   

   elezar

   on Sep 30, 2026

   Member

   More actions

   Thanks @politerealism. I am busy constructing a plan to migrate the e2e/rust tests that are covered by the e2e-podman feature so that we can tackle that in parallel.

   👍React with 👍2Reacted by Polite_realism and Matthew Grossman
6. politerealism

   added a commit that references this issue on Oct 3, 2026

   docs(podman): document and track the e2e-podman rootful coverage gap

   ...

   6015c8e
7. politerealism commented on Oct 6, 2026

   

   politerealism

   on Oct 6, 2026

   Contributor

   More actions

   Update 2026-10-06: Following up on the 2026-09-29 note above where I deferred the e2e/rust→conformance migration to @elezar to avoid duplicating his in-flight work.

   Coordinated directly with Evan today on the 11 general-conformance candidates I'd flagged in a research pass on #3460. He confirmed his own plan (verify under Podman, relax gates on provider_refresh_handles.rs/websocket_conformance.rs, sandbox_labels.rs needs nothing) and suggested migrating applicable ones straight into tests/suites/conformance/ rather than gate-relaxing in place — and invited a partial migration on my end rather than a hard handoff.

   Migrated so far (branch not yet opened as a PR, pending final review on my end):

   • workspace_lifecycle.rs → workspace-lifecycle/workspace-terminating scenarios
   • upload_create.rs → file-transfer/create-upload scenario (grouped with the existing file-transfer family)
   • provider_auto_create.rs → provider-auto-create scenario
   • sandbox_templates.rs → 4 scenarios (lifecycle, get-after-delete, duplicate-name, missing-template)
   • settings_management.rs → settings-management scenario

   All verified against an ephemeral Podman gateway. Will reference #3954 (not this issue) when the PR opens, since that's the umbrella tracking issue for the RFC-0016 testing-strategy work this falls under.

   Two things surfaced along the way worth noting here:

   • port_forward.rs doesn't fit the conformance harness's CLI request/response model (needs a long-lived background SSH-tunnel process + raw socket I/O) — holding that one for Evan's input on whether that pattern belongs on OpenShellRunner or as a scenario-local bypass.
   • Found two of Evan's already-landed file_transfer.rs conformance scenarios (round-trip, git-filtering) currently failing against main — pre-existing, unrelated to this migration, flagged to him separately.
8. politerealism

   added a commit that references this issue on Oct 12, 2026

   test(podman): install script and capture network diagnostics in the e…

   ...

   697565e
9. politerealism commented on Oct 12, 2026

   

   politerealism

   on Oct 12, 2026

   Contributor

   More actions

   Probe results: which excluded e2e-podman binaries run in the tmachine guest (2026-10-12)

   To see how the binaries on podmanE2eFollowUpBinaries actually behave, I used a throwaway draft PR (#4386, not for merge) that lifted 14 of them and ran the rootless Fedora e2e-podman job twice (runs 37982991494 and 38246241655). The second run also installed util-linux-script in the guest and captured network diagnostics on failure. Thanks to Taylor and Simon for running it.

   Run	Tests	Passed	Failed	Skipped
   1	75	66	9	2 (the #[ignore] benchmarks)
   2	75	70	5	2

   Passed in both runs (10 binaries, 14 tests): credential_gating (slowest, about 74 s), forward_proxy_graphql_l7, forward_proxy_jsonrpc_l7, forward_proxy_l7_bypass, host_gateway_alias, landlock, no_proxy, podman_oci_identity, provider_refresh_handles, websocket_conformance. I opened #4422 to stop excluding them.

   sandbox_lifecycle: four of its 19 tests failed in run 1 with spawn openshell sandbox create: No such file or directory, because the PTY helper runs script and the guest doesn't have it. With the package installed, all 19 passed in run 2. #4423 does that.

   Still failing (same five in both runs), and what I could see:

   • driver_config_volume (2 tests): the gateway rejects the create with "caller driver config is disabled". The tmachine gateway config doesn't enable allow_driver_config or enable_bind_mounts; the wrapper scripts do. It looks like a Podman driver-suite fixture, not something to turn on in the shared gateway config.
   • proxy_egress_pipeline (1 of 11 tests): explicit_allowed_ips_and_implicit_ip_literals_succeed_transparently times out on both requests. The other ten pass.
   • transparent_tcp (2 tests): one can't find the musl probe source in the guest (that's test(e2e): restore musl DNS probe coverage for guest prebuilt artifacts #3009), and the other gets an NXDOMAIN from the policy DNS for its fixture hostname.

   For the middle one and the NXDOMAIN, I have a guess, not a conclusion. The diagnostics show the guest has no route to the tmachine bridge subnet (10.89.0.0/24) in its main network namespace. In rootless Podman that bridge lives in Podman's own namespace, so container IPs may not be reachable from where the supervisor runs. If so, both might pass on rootful, which the archive doesn't run today (#4163). A rootful run would tell us.

   Not lifted, on purpose: podman_gateway_start, podman_corporate_proxy, local_driver_token_restart and provider_token_exchange. In tmachine the wrapper-set OPENSHELL_E2E_GATEWAY_* and SPIFFE variables aren't set, so they would skip silently and report a pass. They seem to belong with the disruption and fixture work in RFC 0016.

   Silent skips in tests that already run (found by reading the code): edge_tunnel_e2e's second test and the gateway-restart leg of ssh_host_identity both return early and report a pass, and podman_resource_limits has a redundant driver check. Worth deciding whether to fix or waive them.

   If both PRs land the exclusion list goes from 20 entries to 9. Suggested next steps, happy to adjust: move driver_config_volume into the Podman driver suite with a fixture, and look at the networking guess via #4163. Would it be better to open a separate issue for the rootless networking question, or keep it here?
10. politerealism

    added a commit that references this issue on Oct 12, 2026

    test(podman): explain why each remaining e2e binary is excluded

    ...

    f6f4507
11. matthewgrossman commented on Oct 12, 2026

    

    matthewgrossman

    on Oct 12, 2026

    Member

    More actions

    Heads-up on overlap with the podman_oci_identity entry: #3931 (tracked in #4346, under #2422) moves the OCI USER/WORKDIR checks from the Docker-only custom_image test into a new oci-image feature suite under tests/suites/features. It runs on the Docker rootful, Podman rootful, and Podman rootless tmachine lanes, plus ARM64 Docker.

    Every scenario in that suite also checks that the workload has no capabilities, has no-new-privileges set, and cannot read the sandbox credentials, and that a custom WORKDIR is not a mount. So #3931 trims podman_oci_identity to the checks that need Podman container inspection: the pinned image ID and workload/supervisor isolation.

    A trial merge of #4422 into #3931's current head has no conflicts, and the two fit together: #4422 runs the trimmed podman_oci_identity on rootless tmachine.
12. matthewgrossman commented on Oct 12, 2026

    

    matthewgrossman

    on Oct 12, 2026

    Member

    More actions

    Correction to my comment above: #3931 now also moves the pinned image ID, no-network, credential-mount, and separate-supervisor checks into the shared oci-image suite, so they run on Docker and Podman. podman_oci_identity keeps only the Podman supervisor identity and capability checks. #3931 is ready for review.