Repository navigation
Sandbox exec can exhaust FDs preparing Landlock baseline while remaining Ready #4319
Description
User Story
I use OpenShell to run a long-lived Hermes agent inside a sandbox. I directly encountered a state where the sandbox continues to report Ready, but every new exec request fails because the relay closes before the command reports an exit status. This blocks the agent from starting and makes the sandbox unusable until it is recreated.
Confirmed Root Cause
The capability-free Landlock baseline is prepared before each boundary exec. In the affected implementation, capability_free_baseline_entries() first materializes a Vec<(PathBuf, OwnedFd)>, opening and retaining one file descriptor for every usable top-level filesystem entry before the caller begins adding the rules and releasing those descriptors.
The affected Hermes sandbox image has roughly 987 usable baseline entries. With an OpenShell runtime RLIMIT_NOFILE of 1,024, total descriptor usage can briefly reach approximately 1,023 during exec preparation. The next pipe, socket, or other descriptor allocation can then fail with EMFILE. The SSH execution channel closes without an exit status while readiness continues to reflect the live supervisor/session rather than the ability to admit a fresh exec.
This is a bounded, repeatable per-exec spike, not an unbounded leak over time. Its size is proportional to the number of usable entries in the Landlock baseline. Descriptor usage falls after preparation finishes, so idle or low-frequency sampling can miss the high-water mark.
The earlier DNS/policy-denial sequence was correlated with the first observed failure but is not the root cause. The direct supervisor SSH control remains useful because it proved the failure below Hermes, Omnid, and the gateway.
Impact / Why This Matters
- A valid sandbox image can approach the process FD limit before the requested command starts.
- The client receives
exec relay closedorBroken pipeinstead of an actionable resource error. - The sandbox can remain
Ready, causing callers to retry a runtime that cannot execute commands. - The failure blocks real agent startup even though gateway, supervisor, and sandbox containers remain running.
- Images with more baseline entries have less descriptor headroom and can cross the limit without any workload-level FD leak.
Fix
PR #4324, current head commit 2607df7301614191a38e497babf9baf0693918f0, changes Landlock baseline enumeration to a lazy iterator. Each path is opened, classified, added to the ruleset, and dropped before the next path is opened. It does not change the resulting Landlock policy or grant additional filesystem access.
Measured on the isolated Gen2 VM with the exact patched runtime:
- Patched runtime descriptor high-water across 100 execs: 30
- Patched runtime settled descriptor count: 19
- Direct exec result: 100/100 passed
- Sandbox remained
Readyand accepted fresh execs
The regression test creates 128 baseline entries and asserts that iteration stays within a small constant descriptor margin instead of growing with the number of entries.
Temporary Workaround
Raising the OpenShell sandbox runtime/container nofile limit from 1,024 to approximately 10,000 provides a reasonable temporary safety margin for the currently measured image because the spike is bounded. This is a stopgap for affected deployments, not the permanent fix, and should be reverted after a release containing #4324 is deployed.
The process to observe is the openshell-sandbox runtime. Sampling should run at high frequency while execs are admitted; sampling only an idle supervisor over several hours will not capture the transient preparation spike.
Acceptance Criteria
- Capability-free Landlock baseline preparation keeps descriptor usage bounded independently of the number of baseline entries.
- A sandbox image with approximately 1,000 usable root entries can admit a fresh exec under the default runtime FD limit.
-
openshell sandbox exec <name> -- /usr/bin/truesucceeds repeatedly in the same sandbox. - A failed exec request does not close unrelated SSH channels.
- If the execution boundary is unavailable, the sandbox does not continue reporting usable
Readystate. - Linux regression coverage detects descriptor growth proportional to baseline entry count.
Reproduction / Measurement
- Run the affected OpenShell runtime with
RLIMIT_NOFILE=1024and a sandbox image containing roughly 987 usable top-level entries. - Identify the
openshell-sandboxruntime PID. - Sample
/proc/<runtime-pid>/fdat high frequency while issuing fresh exec requests. Record both count and symlink target type. - Run
openshell sandbox exec <name> -- /usr/bin/truerepeatedly or start the affected Hermes workload. - Observe descriptor usage approach the limit during Landlock preparation and an exec channel close without an exit status.
- Repeat with fix(sandbox): preserve exec capacity under failure and load #4324. Descriptor usage remains near the normal baseline and the exec succeeds.
Environment
- Original OpenShell source:
e7fdd6beef98f7f92d86271a169fdd4d3be44cf3(dev.84) - OpenShell package SHA-256:
c05885dd9d1df34c99eecdb0bcbdd07ebf552a6e26f7da17de277283dca43502 - Supervisor image digest:
sha256:9531e9962dbf56b2dcc9b9f88d8dca2fd99dd652813d4247698fe22e252f498c - Sandbox image digest:
sha256:5b64040c0359ced768f291bf2cbb6c990e8176bc2d564b754807b6142838c799 - Runtime: Docker on Linux/amd64
- Runtime FD limit: 1,024
- Two sequential 4,097-exec runs passed before the real agent-start path; this remains distinct from the lifetime request-ID defect fixed by fix(sandbox): replace lifetime exec cap with retry deadlines #4105/bug(sandbox): long-lived sandboxes stop accepting execs after 4096 requests #4103.
Observed Client Errors
The service is currently unavailable: exec relay closed before the command reported an exit status
channel 0: read failed: Broken pipe
client_loop: send disconnect: Broken pipe
Activity
elezar commentedon Oct 8, 2026
Triage Assessment
Classification: needs-investigation
Summary
The report contains sufficient reproduction, environment, and impact information to validate intake. It describes a credible persistent execution failure with misleading readiness on matched Linux/amd64 Docker components. Source inspection supports the reported SSH failure and readiness mismatch, but does not establish the initiating fault. DNS/network denials are a lead, not a demonstrated cause. No independent runtime reproduction was performed during this assessment.
Investigation
e7fdd6beef98f7f92d86271a169fdd4d3be44cf3, rather than relying on the local checkout or mutable development tags. The direct OpenSSH control narrows the failure to the supervisor execution path or its sandbox backend; it bypasses Hermes, its manager, and the gateway relay.start_exec_specpropagates a backend exec error to the session handler. This can explain a connection ending without an exit status, but the client error alone does not identify the backend fault.boundary_access.session_readiness(), which tracks gateway session acceptance rather than probing fresh boundary execs. A live supervisor/session can therefore remain ready while exec admission fails.PolicyDnsError::Ineligibleto DNSREFUSED.policy_dns_ineligibleconcerns hostname eligibility; DNS alone does not establish executable identity or destination port. Theuvbinary mismatch belongs to subsequent connection enforcement and should be distinguished from an unlisted-host DNS refusal.Boundary gRPC exchange failedis a generic warning carrying anerrorfield. The quoted message omits that field and occurs after the first failures, so it is insufficient to establish the first fault.Relationship to PR #4105 and Related Reports
121930be076427acc408e58ac5e1b31a6202e55f, is an ancestor of the reported source, which is 11 commits later. The GitHub comparison confirms this. The immutable source is more reliable here than the mutabledevrelease metadata.Impact Signals
Evidence Needed for Deeper Investigation
Provide the sanitized effective policy and a minimal script that opens the persistent relay, performs the denied requests, and attempts a fresh exec. Include timestamped supervisor and sandbox logs from before the first failure, preserving nested error details and the boundary warning's
errorfield. Compare persistent relay without denial, denial without persistent relay, and unlisted-host DNS refusal versus a listed endpoint with an ineligible executable. These controls distinguish concurrency, policy path, and transport/recovery failures without assuming a DNS root cause.Human Decision Required
Decide whether to invest in deeper investigation. This assessment records technical validity, not acceptance or sequencing. If accepted, apply
state:accepted, associate the issue with a roadmap item, or both, and decide whether the work remains human-owned. To queue unattended investigation/planning, also applyagent:plan-requested; a direct request to usecreate-spikeorbuild-from-issueseparately authorizes that phase. If declined, close as not planned with the rationale.[-]dev.84: exec boundary permanently fails after policy denials while sandbox remains Ready[/-][+]Sandbox exec can exhaust FDs preparing Landlock baseline while remaining Ready[/+]