[Draft] OSWorld environment on the NeMo Gym sandbox SDK (OpenSandbox runc+KVM)#2079
Draft
terrykong wants to merge 10 commits into
Draft
[Draft] OSWorld environment on the NeMo Gym sandbox SDK (OpenSandbox runc+KVM)#2079terrykong wants to merge 10 commits into
terrykong wants to merge 10 commits into
Conversation
Rebuilt on current main as a minimal, reviewable change set: - nemo_gym/sandbox: endpoint(port) via the provider path proxy + image-less poolRef create (+unit tests) — the only core change (+510/-6) - resources_servers/osworld: sandbox lifecycle via AsyncSandbox, task setup + evaluation in isolated subprocesses over a commit-pinned OSWorld fork - responses_api_agents/nemotron_osworld: host-side Nemotron computer-use loop (prompt/parsing imported from the fork), inference via the gym model server - scripts/sandbox_runtime_poc: k8s deployment assets only (pool, device plugin, qcow2-from-S3, karpenter NodeOverlay, vLLM + durable eval-gate Job) Replaces the previous snapshot branch (tagged pre-slim-03031582): drops the accidental core-file reverts (-2.6k), the dormant UI-TARS agent, and the Slurm driver; +7277/-2658 over 80 files -> +4373/-6 over 47. Signed-off-by: Terry Kong <terryk@nvidia.com>
terrykong
force-pushed
the
osworld-opensandbox-public
branch
from
July 21, 2026 06:20
0303158 to
1806688
Compare
…LD_RUNS_DIR) instead of in-script placeholders Signed-off-by: Terry Kong <terryk@nvidia.com>
…ork without manual setup) Signed-off-by: Terry Kong <terryk@nvidia.com>
…ip install works without GitHub ssh creds) Signed-off-by: Terry Kong <terryk@nvidia.com>
…em (failed install + skip_venv_if_present otherwise wedges the Job) Signed-off-by: Terry Kong <terryk@nvidia.com>
…vers join the head's Ray cluster; mixed versions are rejected at runtime) Signed-off-by: Terry Kong <terryk@nvidia.com>
…s the ray install Signed-off-by: Terry Kong <terryk@nvidia.com>
…ins) for the broken-venv check Signed-off-by: Terry Kong <terryk@nvidia.com>
…herwise scores a solved task 0 as evaluate_exception) Signed-off-by: Terry Kong <terryk@nvidia.com>
Per review: inference is an assumption, not part of the example — the gate takes any OpenAI-compatible POLICY_BASE_URL. Drop the vLLM Deployment and serving assets (model-card recipe covers serving) and the site-specific cell-alias Services. The OSWorld fork pin returns to a placeholder while the fork's public release is decided; the exact pinned state used for all published results is recorded in the internal STATUS/repro doc. Signed-off-by: Terry Kong <terryk@nvidia.com>
Contributor
|
hi @terrykong - flagging another OS world PR here #1408 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A complete OSWorld benchmark environment (
test_nogdrive, 361 tasks) built on thenemo_gym.sandboxSDK, with desktop VMs served by an OpenSandbox pool runningrunc + /dev/kvm (QEMU guest per sandbox — no privileged pods, no VM-per-node):
resources_servers/osworld/— sandbox allocation viaAsyncSandbox(poolRef,image-less create), guest boot/readiness, task setup + evaluation in isolated
subprocesses importing a commit-pinned OSWorld fork, per-rollout verification.
responses_api_agents/nemotron_osworld/— host-side observe→model→act loop fornvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16(public HF); prompt contractand action parsing imported from the pinned fork, inference through the gym model
server (any OpenAI-compatible endpoint).
AsyncSandbox.endpoint(port)(path-proxied guest endpoints) +image-less poolRef create.
scripts/sandbox_runtime_poc/): KVM pool, kvm/tun device-pluginDaemonSet, S3-backed qcow2 PV/PVC (Mountpoint CSI, one read-only image shared by
all VMs), karpenter NodeOverlay for metal autoscale, durable eval-gate Job, and a
Slurm driver (sbatch + README) for clusters that serve the model there.
Results
361/361 tasks, five full runs: 41.8–43.8% aggregate vs the 48.3/48.4% internal
reference runs — including a validation run executed entirely from this branch
(fresh clone, no internal code): 151/361, zero infra failures (run with the then-published fork pin); 8/10 domains inside the reference per-domain band, and the two-seed
union lands in band on the remaining ones. The residual per-run delta is a
load-independent per-task consistency gap (verified across a 4× concurrency range);
full analysis, per-domain tables, and the three evaluation-transport fixes
(vm_file save race, VLC auth-stripping fallback, alien-format guard) are documented
in the repro guide on the branch.
Throughput: full 361-task run in ~2 h wall at concurrency 48; sandbox capacity
autoscales 1→6 metal nodes and back on demand (validated live, zero infra-failure
rows).
Dependency
OSWorld itself is a commit-pinned fork installed via each server's
requirements.txt(repo pattern: mini_swe_agent/tau2). The fork's publicrelease is pending an internal content audit, so the pin is currently a
placeholder — this draft stays draft until the fork (a small delta on
xlang-ai/OSWorld: a no-lifecycleremoteprovider, four evaluator-transportfixes, the Nemotron agent, packaging) is published and re-pinned.
Serving used for the runs
vLLM v0.23.0 (official image) — TP1 on GB300 (k8s Deployment, replicas scale with
eval concurrency) and TP8 on H100 x86 (Slurm, reference-exact shape); recipe flags
per the model card (nano-v3 reasoning parser, xgrammar, mamba cache dtype).
🤖 Generated with Claude Code
Size note: the core delta is the sandbox SDK extension only (+510/−6 incl. tests); everything else is the two new environment servers, their tests/data, and k8s deployment assets — comparable scope to #1377 (+5,963).