chore(monad): merge upstream main 36ee06f7b (before OTLP OCSF export) by mattj-monad · Pull Request #16 · monad-inc/OpenShell

mattj-monad · 2026-10-22T02:25:43Z

Summary

Merge upstream main at 36ee06f7b into monad/main. The last sync was merge upstream main 5e414eadd (OpenShell#10), and this brings in the 37 upstream commits after it.

36ee06f7b is the commit just before export gateway OCSF events as OTLP log records (NVIDIA/OpenShell#4500). This is a routine sync, the first of three stacked PRs. Every fork capability works as it does today. Moving the fork's OTLP log export onto upstream's design is PR 2.

Merge with Create a merge commit. Squash or rebase would drop upstream parentage.

Related Issue

None. This is mechanical upstream maintenance. The next two PRs in the stack bring in export gateway OCSF events as OTLP log records (NVIDIA/OpenShell#4500) and the rest of upstream main.

Changes

There are two commits. The first is a merge commit (--no-ff, parents 2f2786ea0 and 36ee06f7b). The second is a small chart fix.

Conflicts and how they were resolved

Cargo.toml: opentelemetry-otlp now enables the features from both sides. Upstream's gzip-tonic and tls-roots come from enable gzip and https OTLP export (NVIDIA/OpenShell#4399). The fork's logs and tls-webpki-roots stay. opentelemetry_sdk keeps the fork's logs feature.
crates/openshell-otel/src/lib.rs: both sides added TLS for https:// collectors.
- Upstream (NVIDIA/OpenShell#4399) trusts only the platform roots.
- The fork (export gateway and sandbox logs as OTLP log records) trusts only the compiled-in web PKI roots, through tls_config_for, which the fork's log exporter also uses.
- The fork's validated_endpoint and tls_config_for helpers stay. tls_config_for now always trusts the web PKI roots, and adds the platform roots when the platform trust store has at least one certificate.
- Why not upstream's with_enabled_roots(): tonic returns NativeCertsNotFound when native roots are requested and none load. That would break OTLP export on images without a CA bundle, such as the distroless gateway image or a sandbox image running the supervisor.
- Result: everything that works today still works. Upstream's platform-root trust is kept, including SSL_CERT_FILE. That gives the fork something it lacked before: a collector behind a private CA is now reachable.
- openshell-otel now depends directly on the workspace's rustls-native-certs (already in Cargo.lock).
crates/openshell-server/src/grpc/policy.rs: both sides changed the same code in the sandbox-setting delete and upsert paths.
- Upstream's ensure_sandbox_keeps_name check (scope mutation locks and bound lock waits (NVIDIA/OpenShell#4147)) runs first.
- Then the fork's capture of the before and after values for the settings audit event runs.
crates/openshell-supervisor/src/main.rs: upstream's supervisor_exit mapping (shared Docker and Podman suite for image USER and WORKDIR (NVIDIA/OpenShell#3931)) runs first. The fork's final log-push flush (from export agent activity and pin harness versions (OpenShell#7)) runs after it. That order means the "Image workspace validation failed" line is also pushed before process::exit.
deploy/helm/openshell/README.md: both rows are kept. Upstream's securityContext.seccompProfile.type row comes from render Restricted-compatible security contexts (NVIDIA/OpenShell#3340). The fork's server.audit.* rows stay. The order matches what helm-docs generates, and helm:docs:check passes in pre-commit.
docs/how-it-works/gateways/configuration.mdx: the two environment-variable tables are merged.
- Upstream's OTEL_EXPORTER_OTLP_COMPRESSION=gzip row now applies to both traces and logs.
- The fork's log-only transport variables and OTEL_BLRP_* rows stay.
- The TLS sentences on both sides now describe the combined trust behavior. The fork's line "collectors behind a private CA are not yet supported" is removed because it is no longer true.

Semantic fixes inside the merge commit

These files merged cleanly but did not compile or lint together:

policy.rs test: upstream's new AutoApproveChunkContext test from distinguish sibling endpoints and fail closed on solver errors (NVIDIA/OpenShell#4326) did not set the fork's required principal field. It now uses the existing audit_test_principal().
grpc/sandbox.rs: extract portable driver and isolation foundations (NVIDIA/OpenShell#4106) wrote the full path crate::auth::principal::Principal. The fork already imports Principal, so that path triggered the unused_qualifications lint. It is shortened to Principal.

Follow-up commit

fix(helm): render the gateway pod when server.otlp is null. Setting server.otlp: null broke chart rendering, because the collector-auth block from authenticate OTLP log export to a hosted collector (OpenShell#5) read .Values.server.otlp.headersSecret without a default.
- This bug already exists on monad/main: upstream's chart test "treats a null OTLP map as disabled" also fails there (checked with helm unittest on a git archive of origin/monad/main).
- The template now defaults the map before reading headersSecret.

Decisions

No fork capability and no upstream change was dropped.
OTLP TLS trusts the web PKI roots plus the platform roots when there are any (see above). Neither side's behavior was simply picked.
The fork's OTLP log export code is unchanged apart from the shared TLS helper. Migrating it to upstream's design is PR 2.

Testing

All commands were run in the branch worktree:

cargo check --workspace --all-targets: passes after the semantic fixes. The first run failed with E0063 missing field principal, which the policy.rs test fix above resolves.
cargo clippy --workspace --all-targets -- -D warnings: passes.
cargo nextest run -p openshell-otel -p openshell-otel-test-support -p openshell-server -p openshell-supervisor --no-fail-fast: 2,501 passed, 0 failed, 20 skipped. This includes:
- upstream's tls_gzip_export and tls_untrusted_export tests, which check that the platform roots are trusted and that an unrelated CA is rejected;
- the fork's https_endpoint_builds_a_log_exporter_with_public_roots test.
cargo nextest run --workspace --no-fail-fast: 8,306 passed, 1 failed, 32 skipped.
- The failure is openshell-prover runtime_parity::endpoint_selector_single_star_crosses_path_separators. It is the same known failure noted in merge upstream v0.1.4-pre.2 (OpenShell#8) and merge upstream main 5e414eadd (OpenShell#10).
- crates/openshell-prover is byte-identical to upstream 36ee06f7b.
helm lint deploy/helm/openshell: passes.
helm unittest deploy/helm/openshell: 339 of 339 pass after the follow-up commit. Before it, 1 failed, and that failure already exists on monad/main.
mise run pre-commit: passes and leaves no changes.
Not run: e2e tests and a kind deploy.

Checklist

Conventional Commits, signed off
Conflict resolutions described in the merge commit message
Real merge commit; no force push; monad/main history preserved

🤖 Generated with Claude Code

Evaluate issue acceptance using trusted default-branch code and the PR author's repository permissions. Report the native workflow job result, protect enforcement files, and align PR creation and review guidance. Closes NVIDIA#4339 Signed-off-by: Evan Lezar <elezar@nvidia.com>

The VM runtime scripts sourced pins.env optionally and duplicated the LIBKRUN_REF and UMOCI_VERSION defaults, so every bump had to update the same value in several files. Require pins.env, drop the duplicated defaults, and fail on an invalid LIBKRUN_REF instead of silently building upstream HEAD. Environment overrides still take precedence through the defaults in pins.env. Signed-off-by: Florent Benoit <fbenoit@redhat.com>

Signed-off-by: Simon Scatton <sscatton@nvidia.com>

…RKDIR (NVIDIA#3931) * test(oci): share OCI image checks across Docker and Podman Add an oci-image feature testsuite that runs against installed artifacts. It covers custom WORKDIR placement, image content and ownership, workspace writes from the main process and exec, file transfer including directory-merge upload, OCI user identity, the managed /sandbox fallback, and rejection of an unwritable WORKDIR. CI runs the suite on the Docker rootful, Podman rootful, and Podman rootless tmachine guests. Replace the Docker-only custom_image e2e with the shared suite. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): require WorkspaceValidationFailed and cover image USER fallback The unwritable WORKDIR scenario now requires the WorkspaceValidationFailed reason, so unrelated provisioning failures cannot pass it. The numeric USER scenario supplies a policy without a process section, so the image USER is the only source of the sandbox identity. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(podman): leave shared OCI identity checks to the oci-image suite The oci-image feature suite covers the sandbox identity seen by the main process and sandbox exec on Docker and Podman. The Podman e2e keeps the checks only it can make: the pinned image ID and the isolated workload/supervisor container pair. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): accept existing workspace rejection diagnostics Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): fix cleanup ordering and restore coverage parity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): remove migrated legacy workspace helpers and checks Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(drivers): report rejected OCI working directories as WorkspaceValidationFailed The supervisor exits with the reserved status when the sandbox rejects the image working directory. Docker maps that exit during readiness and monitoring and records it before removing the supervisor container, and Podman maps the supervisor companion's exit instead of the workload's. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): require WorkspaceValidationFailed for unwritable WORKDIR Unrelated provisioning failures can no longer pass the rejection scenario. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(oci): keep exit 78 reserved and report fixed workspace diagnostics The supervisor no longer passes on a workload's own exit status 78, so drivers cannot misreport it as a workspace rejection after a runtime restart. The rejection is recognized once at agent start and carried as a typed error. Docker reports the same fixed condition message as Podman and logs the supervisor output instead, and its monitor removes the supervisor container when a stop overlaps diagnostics collection. The OCI image suite registers images for cleanup only after they build. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(oci): simplify workspace failure handling and OCI image tests Report a recorded Docker workspace rejection once, where supervisor readiness fails, and match its fixed message exactly. Read the Podman supervisor exit status once per inspection. Choose the supervisor exit status in one function and derive the workspace error type. The OCI image suite shares step helpers, checks that a custom workspace does not also create /sandbox, and its playbook waits for a connected gateway and collects failure diagnostics through the shared role. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): replace podman_oci_identity with shared and driver suite checks The oci-image feature suite now checks, on Docker and Podman, that the image user has no capabilities, has no_new_privs set, and cannot read the sandbox control credentials, and that a custom WORKDIR is not a mount. The checks that need container inspection (pinned image ID, network=none, supervisor user and capabilities) move to a small test in the Podman driver suite. Remove the legacy podman_oci_identity e2e test. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): keep podman_oci_identity for Podman-only isolation checks Restore the Podman e2e test removed in the previous commit and drop only the checks the shared oci-image suite now runs on Docker and Podman: workload capabilities, no_new_privs, credential readability, and the workspace and /sandbox mount checks. The pinned image ID and the workload/supervisor isolation checks stay, as does the Podman driver suite unchanged. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(oci): simplify workspace failure reporting and OCI image suite Carry the Docker control supervisor's failure reason in a typed error instead of matching status text, build the fixed workspace message where the wait result is classified, and drop the fake Docker server test. Inline image cleanup ordering in the suite, read the container engine once, share sandbox naming, and stop granting a literal /sandbox in the image identity policy. Add e2e:oci-image mise tasks and use them from the ARM64 Docker workflow and TESTING.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): move shared container checks into the OCI image suite Inspect the containers the driver created through the gateway's engine in the shared suite: the workload runs the inspected image ID as the OCI identity without a network or supervisor credential mounts, and the supervisor is a separate non-root container. Keep only the Podman-specific supervisor identity and capability checks in podman_oci_identity. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): check the workspace rejection reason from sandbox JSON Read the WorkspaceValidationFailed condition reason from sandbox get --output json instead of matching CLI output, and restore the Docker driver's original workspace failure helpers. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): capture fixture image identity before sandbox creation Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com>

…le connection (NVIDIA#4452) * fix(kubernetes): restart kube read timeout when each request is written (fixes NVIDIA#4451) Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(kubernetes): use a per-request tower timeout instead of a custom kube client Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…errors (NVIDIA#4326) Signed-off-by: Adrien Langou <alangou@nvidia.com>

…IDIA#4342) Signed-off-by: Adrien Langou <alangou@nvidia.com>

…VIDIA#4422) * test(podman): run the e2e binaries that pass in the tmachine guest Remove ten binaries from podmanE2eFollowUpBinaries so the rootless tmachine e2e-podman archive runs them: credential_gating, forward_proxy_graphql_l7, forward_proxy_jsonrpc_l7, forward_proxy_l7_bypass, host_gateway_alias, landlock, no_proxy, podman_oci_identity, provider_refresh_handles, and websocket_conformance. All ten passed in two consecutive probe runs. Update the exclusion comments and TESTING.md to match what the remaining entries still need. Contributes to NVIDIA#3712. Signed-off-by: politerealism <burdcat17@gmail.com> * test(podman): explain why each remaining e2e binary is excluded Give every remaining podmanE2eFollowUpBinaries entry its own reason, and mark the five that the wrapper-based Podman lane still runs (driver_config_volume, local_driver_token_restart, podman_corporate_proxy, podman_gateway_start, provider_token_exchange) so it is clear they are covered there until tmachine provides what they need. Contributes to NVIDIA#3712. Signed-off-by: politerealism <burdcat17@gmail.com> --------- Signed-off-by: politerealism <burdcat17@gmail.com>

…IA#2441) * refactor(prover): remove legacy unused reachability constraints Remove can_write_to_endpoint() and the solver variables and encoding methods that exist solely to support it. No active query calls this method; confirmed zero callers across the monorepo. Crate is not published externally, so no public API concern. Removed as root cause: - can_write_to_endpoint(): zero callers in the codebase Removed as direct consumers of can_write_to_endpoint(): - binary_can_write: only read inside can_write_to_endpoint() - credential_has_write: only read inside can_write_to_endpoint() Removed as directly dead (no query ever read them): - credential_has_destructive: flagged by compiler, silenced with #[allow(dead_code)] rather than removed - filesystem_readable: flagged by compiler, silenced with #[allow(dead_code)] rather than removed Removed as now-empty encoding methods: - encode_credentials(): exclusively populated credential_has_write and credential_has_destructive - encode_filesystem(): exclusively populated filesystem_readable Also removed two #[allow(dead_code)] allowances and the corresponding HashMap::new() initialisers and build() call sites. Active query behavior is unchanged. All existing prover tests pass without modification and cover all four finding categories end-to-end. No new tests added as they would duplicate existing coverage exactly. Note: removing encode_credentials() leaves write_actions_for_scopes() and destructive_actions_for_scopes() in credentials.rs with no callers. Left as a follow-up outside the scope of this change. Closes NVIDIA#2412 Signed-off-by: Evie Howard <evhoward@redhat.com> * refactor(prover): remove dead write-capability helpers from binary registry BinaryProtocol::can_write(), BinaryCapability::can_write(), and write_mechanisms() lost their only callers when can_write_to_endpoint() and its supporting model fields were removed. Delete the now-dead methods. write_mechanisms() had no callers prior to this series; the two can_write() methods were made dead by the preceding model.rs cleanup. Signed-off-by: Evie Howard <evhoward@redhat.com> * docs(prover): remove stale write-bypass mention from module doc comment The module doc incorrectly listed write-bypass violation detection as a prover capability. The prover only checks reachability queries; write-bypass detection was never implemented. Signed-off-by: Evie Howard <evhoward@redhat.com> --------- Signed-off-by: Evie Howard <evhoward@redhat.com>

* fix(cli): backpressure sandbox uploads Stream archive uploads over a bounded SSH subsystem whose credit follows extraction stdin progress. Preserve the existing input budget and tar destination semantics, validate completion, and clean up cancelled transfers. Add protocol, lifecycle, and large-upload regression coverage without changing russh. Signed-off-by: Shiju <shiju@nvidia.com> * fix(cli): preserve upload cancellation and errors Keep cancellation active through upload setup and retry delays, and terminate cancelled SSH probes. Preserve protocol failures through output completion and report captured diagnostics on early extraction failure. Remove redundant upload validation and extend the affected regressions. Signed-off-by: Shiju <shiju@nvidia.com> * refactor(cli): carry upload credit over SSH streams Replace the framed upload protocol with SSH's own streams. The client sends one JSON plan line, the raw archive and EOF. The supervisor's stdout carries only cumulative totals of bytes written to extraction stdin, extraction output goes to stderr, and the exit status reports the result. Extraction starts inline through the ordinary exec path. This removes the frame codec, SHA-256 digests, the CLI sha2 dependency and the supervisor upload state machine while keeping the 1 MiB credit window, the 4 MiB input guard, cancellation and error reporting. A failed archive now stops SSH before closing stdin, so EOF never marks a truncated archive complete. Signed-off-by: Shiju <shiju@nvidia.com> * fix(cli): isolate upload SSH connections and pin credit cadence - Apply the forward multiplexing overrides to the upload ssh command, so a user ControlMaster cannot carry an upload over another session's master connection. - Kill ssh before closing its stdin when a transfer fails locally, and include ssh stderr in transport errors. - Share ACK_INTERVAL_BYTES from core. Assert at compile time that CLI chunks fit the credit left after an ack and that the window stays below the supervisor's pending-input guard. - Log the real reason when the supervisor rejects an upload. - Test that a single-packet upload reports its final total. Signed-off-by: Shiju <shiju@nvidia.com> * test(cli): assert upload ssh opens the subsystem without a TTY Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com>

* ci(security): temporarily make release Codex scans advisory Signed-off-by: Adrien Langou <alangou@nvidia.com> * ci(security): remove Codex from release scans Signed-off-by: Adrien Langou <alangou@nvidia.com> --------- Signed-off-by: Adrien Langou <alangou@nvidia.com>

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* docs: document workspace and provider labels (fixes NVIDIA#4250) Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * feat(providers): support CLI labels and selector filtering Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(providers): update schema inventory and scoped test environment Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

…2632) Signed-off-by: Evie Howard <evhoward@redhat.com>

) Signed-off-by: Simon Scatton <sscatton@nvidia.com>

…IDIA#4488) * ci(compatibility): review PR breaks and acknowledge release risks Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(ci): use matching compatibility review tooling Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(compatibility): scope Rust SDK review to public exports Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(compatibility): account for Go struct field additions Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(compatibility): restrict egress and fix report retries Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com>

* fix(ci): skip maintainer approval in merge queue Signed-off-by: Jim Meyer <jimeyer@nvidia.com> * refactor(ci): remove maintainer queue-state lookup Signed-off-by: Jim Meyer <jimeyer@nvidia.com> --------- Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

Signed-off-by: Gordon Sim <gsim@redhat.com>

* fix(otel): enable gzip and https OTLP export Signed-off-by: Eric Curtin <eric.curtin@docker.com> * test(otel): reject untrusted OTLP collector certificates Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com>

… and labels (NVIDIA#4497) * feat(kubernetes): add operator allowlist for sandbox pod annotations Add allowed_pod_annotations to the Kubernetes driver config. Sandbox template annotations must match an exact key or a <domain>/* entry, and the default empty list rejects all of them. Keys in the openshell.ai and agents.x-k8s.io domains are always rejected. Expose the setting through the server.allowedPodAnnotations Helm value. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(kubernetes): cover pod annotation rejection on sandbox create Assert the exact rejected annotation keys, exercise create_sandbox_inner with a disallowed annotation, and point the rejection error at the Helm gatewayConfig path. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(kubernetes): drop Helm path from pod annotation rejection error The rejection error reaches API callers, not operators, so name only the gateway config key. Spell the Helm path in the runtimes docs with the quoted table key to avoid an invalid nested gatewayConfig rendering. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(kubernetes): condense pod annotation allowlist tests Fold the allowlist matching and validation cases into table-driven tests, reuse the json_struct helper for templates, and document why the openshell.ai and agents.x-k8s.io domains are reserved. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * feat(kubernetes): add operator allowlist for sandbox pod labels Sandbox template labels were copied onto workload pods unfiltered. Labels select pods for NetworkPolicy rules and Services, so a caller could place a sandbox under rules meant for other workloads or set OpenShell-owned labels. Add allowed_pod_labels, sharing the allowed_pod_annotations entry format, validation, and reserved openshell.ai / agents.x-k8s.io domains. Empty by default, which rejects every template label. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(rfc): propose sandbox service exposure architecture Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(rfc): link service exposure review Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(rfc): shorten service exposure proposal Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(rfc): remove service exposure architecture diagrams Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(rfc): share application ingress across supervisor tenants Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(rfc): describe direct and reverse tunnel proxy connections Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(rfc): propose supervisor service ingress and optional proxy Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(rfc): describe proxy as discovery reference implementation Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(rfc): emphasize supervisor app listener and ingress extensibility Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(rfc): clarify driver-independent service ingress Signed-off-by: Taylor Mutch <taylormutch@gmail.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: Taylor Mutch <taylormutch@gmail.com> Co-authored-by: Taylor Mutch <taylormutch@gmail.com>

* test(server): add a local PostgreSQL test runner The openshell-server tests that need a real PostgreSQL server are ignored by default and had no shared way to run. The mutation-replay test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every contributor had to provision a database by hand, and the advisory-lock tests that follow in this series need the same setup. Add mise run test:rust:postgres. It runs every ignored postgres_* test in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE selects one) on the image pinned by the Kubernetes e2e fixture and removes it on exit. Tests run one at a time because advisory locks are database-wide. The runner always points the legacy replay variable at the selected database, so an inherited URL cannot send that test to a different server. test:postgres-runner checks that selection with a fake cargo and needs no database or container engine. CI does not run the PostgreSQL tests; TESTING.md documents the task. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * fix(server): hold the mutation guard while deleting a provider DeleteProvider checked that no sandbox referenced the provider and then deleted the record without holding the sandbox mutation guard. Sandbox create and provider attach take that guard while they write provider references, so one of them could add a reference after the attached sandbox check passed, and the delete then removed a provider that a sandbox spec still named. Take the guard before the attached sandbox check, as provider create and update already do, so the check and the delete run against a stable set of sandbox references, and reject an empty name after authorization but before the guard. A new test holds the guard, attaches the provider while the delete waits, and asserts that the delete fails with FailedPrecondition and leaves the provider in place. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * fix(server): release staged credentials when a refresh fails early A provider credential refresh stages the minted values under new credential handles before it commits the provider update, and deletes those handles when validation or persistence fails. Two earlier returns skipped that cleanup. The minted expiry was converted to a protobuf timestamp only after staging, so an expiry outside the timestamp range failed the refresh and left the staged values in the credential driver. A failure to acquire the sandbox mutation guard returned the same way. Convert the expiry before staging anything, so a bad value fails before any handle exists, and delete the staged handles when the guard cannot be acquired. A new test refreshes a stored credential with an expiry of i64::MAX and asserts that the call fails, the provider keeps its original handles, and the credential driver holds the same number of values as before. The guard failure path has no test here, because the SQLite guard used by unit tests cannot fail. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): acquire mutation locks on a dedicated PostgreSQL lock pool The cross-object mutation guard took its advisory-lock session from the 10-connection data pool and closed it on every release, so each guarded mutation opened a new database connection, and lock holders competed with their own critical sections for data connections. A lock wait that hit the 10 second lock_timeout surfaced as INTERNAL. Add a lazy, dedicated 4-connection lock pool and a new persistence::mutation_lock module for the lock key, mode and set types. Store::acquire_distributed_mutation_guard now takes a lock set and a deadline, and each lock statement sets lock_timeout to the time left before that deadline, so a cancelled acquisition never waits past it on the server. Returned connections are scrubbed with pg_advisory_unlock_all() and reused, and each return is bounded by 5 seconds so a stalled session cannot keep a pool permit. A lock statement that PostgreSQL ends with SQLSTATE 55P03 fails with a new LockTimeout error, and so does a wait for a lock connection during which every one was checked out at some point, or that started with less than a second left, too little to open one. The API returns it as UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and a 1 second retry delay, because the mutation lock was not acquired, so the request's guarded writes did not run. Only the guard acquisition classifies 55P03; any other 55P03 stays a database error. A lock connection that PostgreSQL does not open by the deadline despite at least a second to do so (refused, out of connection slots, starting up, or not answering, all of which SQLx retries silently) is a database error, not a lock timeout. Mutations still take the legacy global key exclusively, so old and new replicas keep excluding each other during a rolling upgrade. Each PostgreSQL-backed gateway pod can now open up to 14 database connections (10 data plus 4 lock) instead of 10, and the chart's autoscaling.maxReplicas comment says what its default of 4 needs. The SSH host identity lock stays on a data connection, because sandbox creation takes it while holding its mutation guard and a second lock-pool connection there could exhaust the pool. The provider credential refresh lock stays there too, so a refresh still progresses while every lock connection is in use. Part of NVIDIA#3528 Co-authored-by: divesh <dgude@nvidia.com> Signed-off-by: Emilien Macchi <emacchi@redhat.com> * perf(server): partition mutation locks by workspace and sandbox Every cross-object mutation, and every lifecycle and supervisor update, took one process-wide mutex and, on PostgreSQL, one fleet-wide advisory lock. Unrelated sandboxes and providers therefore serialized across all replicas, and one slow critical section delayed every other mutation in the fleet. Replace that lock with hierarchical intention locks on a global key, one key per workspace and one key per sandbox, each held shared or exclusive. Global settings and policy writers and platform-scope profile writers hold the global key exclusively. Provider and workspace-scoped profile writers, including DeleteProvider and credential refresh, hold their workspace key exclusively. Sandbox writers and supervisor reports hold their sandbox key exclusively under shared global and workspace keys. Lifecycle, driver-watch and reconcile paths take only process-local keys and keep relying on compare-and-swap across replicas. Workspace and sandbox keys are domain-separated SHA-256 prefixes. Each guard takes its local keys and then its advisory locks in ascending order under one 10 second deadline, and no path nests guards. The settings mutex is removed, since the global and sandbox keys cover it. Sandbox settings are keyed by sandbox name, which a sandbox key no longer covers once another sandbox reuses the name, so a settings update returns NOT_FOUND unless its sandbox still holds the name after the read, and ABORTED if the settings row was recreated before the write. The global key keeps its legacy value, so older replicas, which hold it exclusively for every mutation, still exclude new ones during a rolling upgrade. Guards now also give up after 10 seconds on SQLite, and provider refresh reports a guard timeout as UNAVAILABLE instead of INTERNAL. A supervisor report for a missing sandbox returns NOT_FOUND. Startup endpoint-status reconciliation still holds the global key for its whole scan. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * fix(server): reconcile endpoint status on startup one sandbox at a time Startup endpoint-status reconciliation held the global mutation key exclusively for its whole scan. On PostgreSQL that blocked every guarded mutation on every replica until the starting gateway had walked all sandboxes, and the scan first waited behind every in-flight sandbox mutation. It also paged with an offset, so sandboxes deleted mid-scan shifted rows between pages, and a write that raced the scan (a Conflict from a lifecycle update on another replica, or a sandbox deleted after it was listed) failed gateway startup. Walk the sandboxes with keyset paging instead, and reset each candidate under its own sandbox-scoped guard, up to four at a time to match the lock pool. A candidate with a fresh shared owner is skipped before any guard is taken. Under the guard the reconciliation re-checks the owner, re-reads the sandbox, skips it when it is gone or no longer carries endpoint status, and writes against the version it just read. A Conflict or a database error at write time, which is how a row deleted after the re-read surfaces, is retried with a fresh read, at most five attempts per sandbox. A lock timeout on the guard is retried the same way, so a holder of the sandbox, its workspace or the global key that releases within the retry window cannot stop a replica from starting. No lock-timeout retry starts once the scan has run for three lock timeouts, so a scan stuck behind a long holder fails within about 40 seconds, inside the chart's 60-second startup probe, and logs the sandbox it gave up on. Store::list_all_messages loses its only caller and is removed. Reconciliation stays fail-closed: a holder that outlasts the retry window, an owner lookup failure, or a sandbox that keeps changing still fails startup, so stale endpoint success is never served. User-visible change: a restarting gateway no longer stalls mutations of unrelated sandboxes and providers on its peers while it reconciles. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * perf(server): skip the endpoint-status lock when a session was replaced When a supervisor session ends, the gateway resets that sandbox's tool server endpoint evidence unless a replacement session already owns endpoint observation. The replacement check ran only after the reset had read the sandbox and taken its sandbox mutation guard, so a reset whose supervisor had already reconnected still queued behind other mutations of that sandbox, and on PostgreSQL it used a lock-pool connection, only to find it had nothing to do. Check for a replacement first: a live local session, or a fresh owner record on a peer. When one exists, return without reading the sandbox or taking the guard. Otherwise take the guard and repeat the same check under it, because a replacement's pre-acknowledgement reset runs under the same sandbox key and resetting after it would wipe the replacement's fresh evidence. Both checks share one helper, so they cannot drift apart. When no replacement exists, the reset costs one extra owner-row read. This helps most when a replica notices a dead stream late, after the supervisor has already reconnected elsewhere (a network partition or a keepalive expiry). Deferring to a replacement leaves the old session's evidence to the replacement's pre-acknowledgement reset, which can time out on its bounded mutation lock wait. That reset's failure path removed the replacement and released its owner record but scheduled no invalidation, so the old session's success evidence could outlive both sessions. An old session that ended after its replacement registered already deferred this way, and the unguarded check now makes every disconnect that finds the replacement defer too. When the reset fails while the replacement is still current, also schedule the retried invalidation that follows a session end. That failure path keeps the lifecycle demotion from abandon_session_setup, but runs it in the spawned task before the retry and only while this session is still current, so a timed-out reset returns promptly. Part of NVIDIA#3528 Co-authored-by: divesh <dgude@nvidia.com> Signed-off-by: Emilien Macchi <emacchi@redhat.com> * test(server): cover staged credential cleanup on a lock timeout A provider credential refresh deletes its staged handles when it cannot acquire the provider mutation guard, but that path had no test: the SQLite guard used by unit tests could not fail. Mutation locks now acquire under a deadline that tests can shorten, so the timeout path can be exercised directly. The new test shortens the lock deadline to 50 ms, holds the workspace mutation guard, and refreshes a provider that already stores an AWS access key. The refresh returns UNAVAILABLE with the MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle, which still resolves to the old key, and the credential driver holds the same number of values as before the refresh. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): export mutation lock wait and timeout metrics A handler mutation guard can now wait up to 10 seconds for its keys and then fail with UNAVAILABLE, but nothing showed operators how long guards waited or how often they gave up, short of reading warning logs on each replica. Export openshell_server_mutation_lock_wait_seconds, a histogram of the time one guard took to acquire all of its keys (local registry plus PostgreSQL advisory locks), recorded on success, and openshell_server_mutation_lock_timeouts_total, a counter of guard acquisitions that timed out. Both carry a bounded scope label: global, workspace or sandbox. The platform scope, whose workspace is empty, counts as global. The histogram uses the same 1 ms to 15 s buckets as the relay and peer histograms, and the timeout counter starts at 0 for every scope, so rate() works on an idle replica. The timeout warning now also logs the scope. A guard that fails for another reason, such as a lock connection PostgreSQL did not open, is not counted as a timeout and logs a "mutation lock acquisition failed" warning instead. Lifecycle, driver-watch and reconcile paths take only process-local locks and never time out, so they are not recorded. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * test(server): add PostgreSQL mutation lock integration tests The scoped mutation locks were only unit-tested on SQLite, where the database layer is a no-op. Add ignored postgres_* tests that run the advisory-lock layer against a real PostgreSQL through mise run test:rust:postgres, each in its own disposable schema (TestSchema). persistence/mutation_lock_pg_tests.rs uses two stores on one database as two replicas. It covers disjoint sandbox scopes holding at once, a workspace key blocking only sandboxes in that workspace, the global key blocking every scope, and exclusion in both directions against a raw holder of the legacy global key, whose 55P03 stays a database error outside a guard acquisition. It checks that timed-out and cancelled acquisitions release the keys they already took, that a released lock connection returns to the pool without residual locks, that a release stalled on the network closes its session and the pool recovers, and that the lock pool never grows past its size. It also checks that a full lock pool times out as lock contention, while a lock connection that PostgreSQL never answers fails as a database error. compute/mutation_guard.rs runs the random scope mix across two PostgreSQL runtimes and checks that a by-id guard waits for a workspace writer on the other replica. The provider refresh lock test moves onto the runner too, and checks that a refresh proceeds while every lock connection is held. A bench_postgres_* capacity check measures a paced burst of 500 reconnects 12 ms apart into one replica and fails when the p99 lock wait reaches a fifth of the lock timeout. Machine load moves those waits, so mise run test:rust:postgres skips it and mise run test:rust:postgres:bench runs it alone, through a new OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner. The tests add test-only helpers: the lock pool's size and idle count and the backend pid of a held lock session. Nothing changes outside tests and the test runner. Part of NVIDIA#3528 Co-authored-by: divesh <dgude@nvidia.com> Signed-off-by: Emilien Macchi <emacchi@redhat.com> * ci: run the PostgreSQL-backed server tests Branch Checks now runs the postgres_* tests through the same runner developers use, on an x86_64 Linux runner with Docker. They cover lock scope exclusion, interop with replicas still on the old global key, cancellation, lock-pool bounds and connection failures, which no other CI lane checks directly. The load-sensitive bench_postgres_* capacity checks stay local. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * docs: document scoped mutation locks and lock-pool sizing In the High Availability guide, size PostgreSQL for 14 connections per gateway pod, 10 for data and 4 for the new lock pool, so three or more replicas now exceed what the PostgreSQL default leaves, and keep headroom for lock sessions that cancelled requests leave behind. Explain which changes lock a sandbox, a workspace, or the whole fleet, and that a lock wait over 10 seconds returns a retryable UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to raise max_connections to the 14-per-pod size before upgrading, because the rollout already runs new pods at that count, and warns that older replicas keep serializing every mutation until the rollout finishes. Monitor Capacity gains the lock metrics and their queries, and names slow driver, credential, middleware, or profile source calls as another cause of lock waits. List the lock wait histogram and timeout counter, with their scope label, on the Gateway Metrics page. The counter also counts background work and leaves out lock connections that PostgreSQL does not open although at least a second of the wait remained. In the cluster debugging skill, replace the single advisory-lock paragraph with the lock levels and pool, what each timeout detail means, how a lock connection that PostgreSQL does not open shows up, when terminating a lock backend is safe, a pg_locks query that shows holders and waiters, and how to spot the global key. The per-pod greps also read the timeout counter, and new rows cover lock timeouts and lock connections that PostgreSQL does not open. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): check mutation lock ordering in debug builds The mutation lock ordering rules were prose only, so a nested guard on a different key passed every test and could only deadlock under contention. Debug builds now register each held guard with the Tokio task (or the thread, outside a task) that acquired it, and panic before waiting when a local key, a blocking lifecycle gate or the SSH identity lock would break rules 1, 4 or 5. Startup endpoint-status reconciliation and the tests that hold a guard while driving guarded code run under lock_order::branch(), which marks deliberate concurrency within one task. Release builds compile the check out. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * fix(server): do not hold gateway shutdown for the endpoint-status retry When a replacement supervisor session failed its endpoint-status reset, the spawned cleanup task kept the session lifetime guard through retry_endpoint_status_after_supervisor_disconnect. That retry only ends once the reset is durable, so a gateway shutdown during a database or lock outage waited the full 10 s and reported ownership cleanup as incomplete. The guard now covers only the lifecycle demotion, as in finish_supervisor_session, and a SQLite test checks that shutdown completes while the retry is still blocked. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * refactor(server): make the platform profile lock scope explicit Platform-scope provider profile writers (import, update, delete) pass an empty workspace on purpose and need X(global), because they validate sandboxes in every workspace. MutationScope::profiles() now names that escalation and picks Global for the platform scope, instead of relying on MutationScope::Workspace("") to mean it. An empty workspace name now trips a debug assertion, and release builds keep X(global) as a fail-safe that over-locks rather than under-locks. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * perf(server): amortize local mutation lock table sweeps LocalMutationLocks swept its whole table on every key lookup, so a burst of N concurrent acquisitions on one replica cost O(N^2) under a blocking mutex. The table is now swept only when a new key grows it past the larger of 64 and twice the live entries left by the last sweep, which makes acquisition amortized O(1). Released entries, including those of cancelled waiters, stay bounded at about twice the peak live set. The lifecycle gate registry keeps its per-call sweep. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): make the database pool ceiling configurable Both stores hardcoded their pool ceiling: 10 connections for Postgres, 5 for on-disk SQLite. One pool is shared by every database-backed RPC, so that number is also the gateway's ceiling on concurrent database work. Once every connection is checked out, callers queue on acquire and sqlx logs "time to acquire exceeded slow threshold"; sandbox creates then time out and are retried, which adds load rather than shedding it. The right ceiling is deployment-specific — it depends on how many gateway replicas share the database and what max_connections the server itself allows — so it cannot be a single number baked into the binary. Expose it on the same config surface as the rest of the gateway's settings: --db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS, and the TOML key database_max_connections, resolved in that precedence order. Helm deployments set the key through gatewayConfig. Omitting it keeps each backend's previous value, so existing deployments do not move. Values below 1 are rejected rather than silently replaced by the default: a zero pool would block every acquire, and quietly ignoring a typo would reproduce the ceiling the operator is trying to lift. An in-memory SQLite database stays pinned to one connection, since the database lives inside that connection. Refs: NVIDIA#2561 Signed-off-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com> (cherry picked from commit 33517b3) [emacchi: dropped the server.dbMaxConnections Helm value; operators set the key through gatewayConfig] Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): make the mutation lock pool size configurable The mutation lock pool was fixed at 4 connections per replica. That number rests on one reconnect-burst bench rather than scale testing, so a gateway that serves many more sandboxes needs a way to raise it without a rebuild. Add --db-lock-max-connections, OPENSHELL_DB_LOCK_MAX_CONNECTIONS and the database_lock_max_connections gateway.toml key next to the data pool knob, resolved in the same order; Helm deployments set it through gatewayConfig. The default stays 4, zero is rejected, SQLite ignores it with a warning, and the gateway logs both pool sizes at startup. PostgreSQL now needs at least 2 data connections, because the SSH identity lock keeps one while it queries the pool. The connection sizing docs, the HA guide formula and the debug skill now count data plus lock connections. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): export mutation lock errors and lock connections in use A mutation lock acquisition that failed without timing out, such as a lock connection PostgreSQL never opened or a failed lock statement, only logged a warning, and readiness stays healthy because it pings the data pool. Count those failures in openshell_server_mutation_lock_errors_total{scope}, kept apart from timeouts because they point at PostgreSQL capacity or connectivity rather than contention. Export openshell_server_mutation_lock_connections_in_use as a gauge slot held by each checked-out lock connection, next to openshell_server_mutation_lock_connections_capacity, which reads the configured lock pool size; both exist only with PostgreSQL. The metrics catalog, the HA guide's signals and PromQL examples, and the debug skill cover the new series. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): time mutation lock holds and test a stalled holder Nothing bounds how long a mutation guard is held, so a holder stuck in a compute driver, credential backend or middleware call keeps its scope locked while waiters time out. Each guard now records its hold in openshell_server_mutation_lock_hold_seconds{scope} when it is released, and logs a warning when the hold outlasted the lock wait timeout, so operators can find the replica and scope behind those timeouts. A PostgreSQL test with a stalled holder pins the current contract: waiters on its sandbox, workspace and global scopes fail by their deadline, other scopes proceed, the holder keeps its key, and a full lock pool on one replica leaves the other working. The metrics catalog, the HA guide and the debug skill describe the new signals, which observe holds without bounding them. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> --------- Signed-off-by: Emilien Macchi <emacchi@redhat.com> Co-authored-by: divesh <dgude@nvidia.com> Co-authored-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>

) * fix(compute): allow stopping errored sandboxes for recovery Signed-off-by: Shiju <shiju@nvidia.com> * test(compute): focus sandbox error recovery coverage Remove duplicate cancellation and idempotency checks covered by existing lifecycle tests. Keep recovery guidance focused on the stop/start commands and timeout exception. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com>

…#2449) The core IPv6 forwarding fix landed on main via NVIDIA#2552; this carries the docs example forward into the current Port Forwarding section of docs/how-it-works/sandboxes/overview.mdx after the page move in NVIDIA#3705. Remaining scope is documentation only. Signed-off-by: Andrew White <andrewh@cdw.com> Co-authored-by: andrewwhitecdw <andrewwhitecdw@users.noreply.github.com>

Signed-off-by: Eric Curtin <eric.curtin@docker.com>

…VIDIA#4106) * refactor(runtime): decouple supervisor access and boundary audit validation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): keep supervisor process access boundary driven Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(cli): support environment values from caller variables Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): extract driver-independent MXC foundations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): defer UI policy support to MXC integration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): integrate current policy and SSH launch contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): reconcile fixtures with updated main Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): repair stacked branch CI contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): update boundary fixtures for current contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(runtime): clarify scoped guards for launch inputs and delete Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): validate foreground attachment intent before launch Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>

…A#4534) * ci: use lifecycle:frozen as the sole stale issue exemption Signed-off-by: Evan Lezar <elezar@nvidia.com> * ci: support state:frozen as a stale issue exemption Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com>

* Add triage duty tooling Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(triage): surface dependabot prs early Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(triage): keep local state untracked Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(triage): clarify queue continuation order Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(triage): remove playbook whitespace Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(triage): address duty tooling review feedback Signed-off-by: Jim Meyer <jimeyer@nvidia.com> * docs(triage): minimize changes to original duty skills Signed-off-by: Jim Meyer <jimeyer@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Jim Meyer <jimeyer@nvidia.com> Co-authored-by: Jim Meyer <jimeyer@nvidia.com>

The seccomp notification probes install their listener on a launcher thread and receive it over a channel. When the install fails, the launcher drops the sender and exits, so the broker only sees a closed channel and returns a generic "launcher disappeared" error. The thread is never joined, so the kernel's errno is discarded. That collapses several distinct failures into one opaque string: a notification struct size mismatch, a failed PR_SET_NO_NEW_PRIVS, and any non-EINVAL SECCOMP_SET_MODE_FILTER rejection all look identical to a pre-5.19 kernel refusing WAIT_KILLABLE_RECV, which the install path already retries without the flag. Join the launcher on the recv() failure path and propagate its error unchanged, so both the ErrorKind and raw_os_error survive to the qualification report. Apply this to all three probes. Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

…A#4263) * fix(supervisor): use jsonl files for independent OCSF retention Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(supervisor): trim and separate log retention regressions Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(supervisor): drop legacy OCSF log migration Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(observability): remove legacy OCSF filename guidance Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(ocsf): remove stale supervisor log filename comments Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(supervisor): clarify denial reader log source Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(supervisor): retain OCSF JSON opt-in explanation Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Signed-off-by: Yuedong Wu <dwcn22@outlook.com>

…NVIDIA#4521) Add two scenarios to the shared oci-image feature suite: - An image without USER, with no policy identity, runs as the numeric default 1000:1000 without the supplementary groups of the image's UID 1000 account. - A policy run_as_user and run_as_group replace an image's USER root. Both check the main process, sandbox exec, and the workload container user. Factor policy temp files into a small helper. Closes NVIDIA#4431. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…rkspace checks (NVIDIA#4527) In operator workspace mode the namespace label watcher fills the allowlist in the background after the driver starts. Workspace checks that ran before its first full listing rejected labeled namespaces with PermissionDenied, so provider refreshes due at gateway startup were parked as configuration errors for an hour. Track whether the allowlist has received an authoritative listing. Operator workspace checks now wait up to 10 seconds for it and return Unavailable if it is still missing, which callers such as the provider refresh worker retry shortly. An unreadable namespace file still yields a listed, empty allowlist that fails closed. Closes NVIDIA#4526 Signed-off-by: Mrunal Patel <mrunalp@gmail.com>

The core, server, and sandbox-backend build scripts always used the vendored protoc binary and ignored PROTOC, so a distro build could not use its system compiler. When PROTOC is set, use it and let prost resolve the includes (PROTOC_INCLUDE still applies); otherwise behave exactly as before. Contributes to NVIDIA#3510. Signed-off-by: politerealism <burdcat17@gmail.com>

…A#4468) rcgen 0.13.2 declares its optional x509-parser dependency with the verify feature, which pulls in ring. The workspace enabled rcgen's x509-parser feature for every crate, so the CLI and gateway linked ring alongside aws-lc even though nothing in them verifies signatures with x509-parser. Only SandboxCa::from_pem in openshell-supervisor-network needs CertificateParams::from_ca_cert_pem. Enable the feature there alone. ring no longer appears in the dependency trees of openshell, openshell-gateway or openshell-prover; it remains in openshell-supervisor until rcgen is upgraded. Release builds compile one package at a time, so the feature does not leak into the other binaries. Signed-off-by: politerealism <burdcat17@gmail.com>

Bumps [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) from 4.37.7 to 4.38.3. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@ff2f1c6...24c5418) --- updated-dependencies: - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.38.3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…K module (NVIDIA#4439) * fix(python): let proto stub generation skip .pyi for older protoc Add a --no-pyi option to generate_python_proto.py for toolchains whose protoc predates --pyi_out, such as Fedora's grpcio-tools 1.48, and list python3-googleapis-common-protos and python3-httpx in the spec's Recommends and dist-info Requires-Dist, which the SDK needs to import. Default behavior is unchanged. Contributes to NVIDIA#3510. Signed-off-by: politerealism <burdcat17@gmail.com> * fix(rpm): install every SDK module and record the license accounting The python3-openshell %install copied only __init__.py, sandbox.py and _proto/, but the package imports .errors, .mutations and .providers and ships a py.typed marker, so an installed RPM could not import openshell. Install the missing files and add a %check that fails if any relative import in the installed package has no matching installed module. Importing the package needs grpcio, which the build environment lacks, so the check is static. Set License to the minimal SPDX expression covering every crate linked into the shipped binaries, keep the pure-Python subpackage at Apache-2.0, and note how the non-crate content is covered. Signed-off-by: politerealism <burdcat17@gmail.com> --------- Signed-off-by: politerealism <burdcat17@gmail.com>

Merge upstream main up to 36ee06f, the commit just before NVIDIA#4500 (export gateway OCSF events as OTLP log records), 37 upstream commits since the last sync at 5e414ea. Conflicts: - Cargo.toml: opentelemetry-otlp takes the union of features. Upstream's gzip-tonic and tls-roots (NVIDIA#4399) join the fork's logs and tls-webpki-roots; opentelemetry_sdk keeps the fork's logs feature. - crates/openshell-otel/src/lib.rs: keep the fork's validated_endpoint and tls_config_for helpers, which the fork's log exporter shares. TLS now trusts the compiled-in web PKI roots always and adds the platform trust store when it has any certificate. That keeps the fork working on images without a CA bundle (tonic refuses native roots when none load) and gives upstream's platform-root behavior, including SSL_CERT_FILE for a private CA. - crates/openshell-server/src/grpc/policy.rs: keep upstream's ensure_sandbox_keeps_name check (NVIDIA#4147) ahead of the fork's settings audit capture in both setting delete and upsert paths. - crates/openshell-supervisor/src/main.rs: map the exit status with upstream's supervisor_exit (NVIDIA#3931) first, then run the fork's final log push flush, so a workspace validation failure line is pushed too. - deploy/helm/openshell/README.md: keep upstream's securityContext.seccompProfile.type row and the fork's server.audit.* rows. - docs/how-it-works/gateways/configuration.mdx: merge both env var tables (gzip compression for traces and logs, log-only transport vars, OTEL_BLRP_*), and describe the combined TLS trust. Semantic fixes in the merge: - policy.rs test: upstream's new AutoApproveChunkContext test gets the fork's required principal field. - grpc/sandbox.rs: upstream's fully qualified Principal path trips unused_qualifications because the fork already imports Principal. Signed-off-by: Matt Jane <matt@monad.com>

The OTLP collector auth block dereferenced server.otlp.headersSecret directly, so setting server.otlp to null (the documented way to disable OTLP, covered by the 'treats a null OTLP map as disabled' chart test) failed to render. Default the map before reading headersSecret. Signed-off-by: Matt Jane <matt@monad.com>

elezar and others added 30 commits October 13, 2026 11:44

fix(release): qualify prerelease tag pushes (NVIDIA#4335)

a9a635a

Signed-off-by: Simon Scatton <sscatton@nvidia.com>

fix(prover): distinguish sibling endpoints and fail closed on solver …

c608c13

…errors (NVIDIA#4326) Signed-off-by: Adrien Langou <alangou@nvidia.com>

fix(vm): bind cached registry images to verified manifest digests (NV…

ea74f2b

…IDIA#4342) Signed-off-by: Adrien Langou <alangou@nvidia.com>

fix(sandbox): release broker-owned listening sockets (NVIDIA#4444)

fce4e5a

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

refactor(prover): separate outcome evaluation from rendering (NVIDIA#…

0d4d22e

…2632) Signed-off-by: Evie Howard <evhoward@redhat.com>

fix(sdk-ts): preserve rpc errors for out-of-range responses (NVIDIA#4495

b9dc1f6

) Signed-off-by: Simon Scatton <sscatton@nvidia.com>

fix(core): retry transiently busy e2fsprogs probes (NVIDIA#4315)

2f6c691

Signed-off-by: Gordon Sim <gsim@redhat.com>

docs(cli): document sandbox create size limits (NVIDIA#4447)

39b13d1

Signed-off-by: Eric Curtin <eric.curtin@docker.com>

lunarwhite and others added 9 commits October 14, 2026 17:16

fix(helm): render Restricted-compatible security contexts (NVIDIA#3340)

3f66c26

Signed-off-by: Yuedong Wu <dwcn22@outlook.com>

Navigation Menu

Repository navigation

chore(monad): merge upstream main 36ee06f7b (before OTLP OCSF export) - #16

chore(monad): merge upstream main 36ee06f7b (before OTLP OCSF export)#16
mattj-monad wants to merge 39 commits into
monad/mainfrom
sync/upstream-main-2026-10-21-1-pre-otlp

mattj-monad commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

Conversation

mattj-monad commented Oct 22, 2026

Summary

Related Issue

Changes

Conflicts and how they were resolved

Semantic fixes inside the merge commit

Follow-up commit

Decisions

Testing

Checklist

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants