feat(harbor_agent): NeMo Gym sandbox execution backend + Terminal-Bench 2.1 config

[hemildesai]

What

Integrates the nemo_gym.sandbox API into the harbor agent non-intrusively: a new NemoGymSandboxEnvironment (a Harbor BaseEnvironment) executes Harbor trials inside sandboxes managed by any NeMo Gym sandbox provider (opensandbox, docker, daytona, apptainer, ecs_fargate). Selected purely via config — no changes to the harbor_agent execution path.

• custom_envs/nemo_gym_sandbox/environment.py — AsyncSandbox-backed backend: task.toml resources → SandboxSpec, tar-based upload_dir/download_dir with per-file fallback, bash -ic exec wrapping (parity with Harbor's docker/daytona backends), always-terminate on stop(), opt-in allow_unenforced_internet_isolation, image_rewrites.
• configs/harbor_agent_opensandbox.yaml — Terminal-Bench 2.1 (89 tasks) with stock Terminus-2 through the opensandbox provider in server-proxy mode. Provider referenced via sandbox_provider: ${sandbox} (mini_swe_agent_2 style; include the shipped provider config with a second --config). Sampling/harness match the Qwen3.6-27B model card (temp 1.0 / top_p 0.95 / top_k 20, 3h agent timeout, 256K ctx / 80K max output). 30 GiB sandbox ephemeral storage to avoid kubelet evictions on disk-heavy tasks.
• prepare_terminal_bench_2_1.py + checked-in 89-row benchmark input JSONL (pinned harbor-framework/terminal-bench-2-1).
• harbor_ray_task_num_cpus config knob — Harbor jobs are I/O-bound Ray tasks; default 1 CPU/task caps concurrency at the driver CPU count, so a fractional value (0.25) enables true full-89 concurrency on a 24-CPU driver.
• fix: extract_usage sets input_tokens_details.cache_write_tokens (required by openai ≥2.40; every live-model /run 500'd without it).
• 18 unit tests (fake registered provider; tar round-trip verified at byte level).

Result — Terminal-Bench 2.1, Qwen3.6-27B

Ran the full 89-task benchmark end-to-end: vLLM-served Qwen3.6-27B on 2 GPU replicas, gym/Ray driver, and sandboxes on a live OpenSandbox deployment reached over an istio ambient multi-cluster mesh (server-proxy mode, no public endpoint). Traces captured for all 89 tasks (rollouts + per-trial trajectory, asciinema recordings, verifier artifacts).

	Value
This run (TB 2.1, Harbor/Terminus-2, single run)	pass@1 = 59.55% (53/89)
Model card (TB 2.0, Harbor/Terminus-2, avg of 5 runs)	59.3

Reproduces the model-card Terminal-Bench number essentially exactly (within ~0.25 points). Caveats: TB 2.1 vs 2.0 (26 tasks fixed; same count/harness), single run vs 5-run average, and sandbox resources set to 4 vCPU / 16 GiB / 30 GiB.

Note: commits carry DCO sign-off; GPG signature to be added on amend (non-interactive session).

🤖 Generated with Claude Code

Update: cross-cluster hardening + 5-run measurement (in progress)

Follow-up commit hardens the opensandbox provider + harbor agent for sandboxes reached across a load-balanced/multi-cluster path, based on failure modes hit while running the benchmark end-to-end:

• connection.disable_connection_pooling (provider, on by default in the shipped config): a load balancer between client and OpenSandbox server can silently reap idle pooled connections; the SDK then reuses a dead socket and exec hangs (observed as dozens of CLOSE_WAIT sockets wedging tasks, while fresh-connection requests completed in ~1s). With pooling disabled every request opens a fresh connection — no reaped socket is ever reused. Verified A/B on a live mesh: identical run with pooling on collapsed (8/89 tasks in 40 min, CLOSE_WAIT≈80); with the option on, 57/89 in 27 min with CLOSE_WAIT=0.
• Hard exec wall-clock cap (provider): backstop around submit+poll+retries so a pathologically wedged exec raises TimeoutError (which Terminus-2 records as a command timeout) instead of hanging the task.
• default_exec_timeout_s 14400 → 300 — superseded: that reasoning accounted only for Terminus-2's tmux plumbing and missed Harbor's verifier, which also calls exec() without timeout_sec. The value is now 1800 in both the env default and the TB config — see the gold-patch section below for the measurement.
• nemo_model_server_timeout_sec 7200 → 2400: lets a silently-dropped model call self-heal within 40 min instead of blocking a task for 2h.
• cpu_pin_enabled (opt-in, default false): pins every Harbor exec (tmux session + verifier) to a random contiguous core block sized by the task's cpu count, so nproc inside reports the real budget and co-resident sandboxes spread across core blocks. Fail-open (no taskset/small host → unpinned). Measured on a full TB 2.1 run under card conditions: no e2e-time or score delta vs baseline (58.4% / 3h13m / 12.4% timeouts, all within the 5-run baseline spread) — task wall time on this benchmark is dominated by model turns, not sandbox CPU contention. Kept for CPU-bound workloads.

Serving guidance for Qwen3.6 (hybrid GDN attention) on vLLM, learned the hard way:

• Use stock vllm/vllm-openai (v0.25.1 used here; multi-arch incl. arm64).
• Disable prefix caching (--no-enable-prefix-caching) — superseded: the engine-core freezes initially attributed to prefix caching turned out to be the default (FlashInfer) GDN prefill kernel; with --gdn-prefill-backend triton, prefix caching ON is stable (validated over a further 5-run cohort, zero freezes) and scores better — see the final card-parity section below.
• Warm each replica before an eval burst (a short + a long generation crossing the ~8k boundary, plus a small concurrent batch) to pre-trigger kernel JIT.

5-run pass@1 (final)

Two cohorts, differing only in the per-task agent timeout (the 4h budget compensates the per-turn re-prefill slowdown from running without prefix caching — see serving notes):

Run	Agent timeout	pass@1	agent-timeout rate
1	3h	57.30% (51/89)	16.9%
2	3h	53.93% (48/89)	20.2%
3	4h	62.92% (56/89)	10.1%
4	4h	53.93% (48/89)	11.2%
5	4h	59.55% (53/89)	10.1%

• 3h cohort mean: 55.6 · 4h cohort mean: 58.8 ± 4.5 · all-5 mean: 57.5 ± 3.8
• References: earlier single clean run 59.55%; model card 59.3 (TB 2.0, 5-run average)
• The 4h cohort — whose timeout rate matches the card-harness conditions (~10%) — lands on the model-card number within noise; run-to-run spread of ±4-5 points on an 89-task benchmark is expected (1 task ≈ 1.1 points).

Serving notes for Qwen3.6 (hybrid GDN) on vLLM

Hard-won operational guidance from ~30h of serving under full-benchmark agentic load:

• --gdn-prefill-backend triton — with the default (FlashInfer) GDN prefill kernel, the engine core froze roughly hourly under concurrent long-context load: requests parked, prompt/generation counters static, no errors, health endpoint green. Reproduced on two different vLLM builds, with prefix caching on and off, on fresh and long-lived engines. On the Triton/FLA kernel: zero freezes across ~16h of identical load. (An upstream issue with the full signature is worth filing.)
• Prefix caching: keep it ON once the Triton GDN kernel is in place. It was temporarily disabled while isolating the freeze (it's flagged experimental for this hybrid-GDN family), but the final card-parity cohort ran 5 freeze-free runs with caching enabled — and it's worth ~3–4 points on this workload: per-turn re-prefill dominates deep agentic trajectories, so caching cuts turn latency, which cuts agent-timeout rate (~11% vs ~13–17% cacheless) even on a shorter 3h budget.
• --compilation-config '{"cudagraph_mode": "PIECEWISE"}' — attention ops run eagerly, outside graph replay.
• Warm each replica before an eval burst (one short + one long generation crossing the ~8k boundary + a small concurrent batch) to pre-trigger kernel JIT; and restart engines between long evals.
• Pin eval driver pods against node consolidation (karpenter.sh/do-not-disrupt: "true" or equivalent) — multi-hour drivers are casualties of routine autoscaler bin-packing otherwise. The same applies to sandbox pods on autoscaled pools: one mid-eval node roll (triggered by an AMI-alias drift) killed ~12 in-flight sandboxes whose pods carried no disruption protection, surfacing as opaque proxy 500s.
• Sandbox resources 8 CPU / 16 GiB / 30 GiB for TB 2.1's heavier tasks.

Follow-up: 4-replica scaling + SGLang A/B (5 runs each, 4h timeout, 8C/16G sandboxes)

Serving stack	pass@1 (5 runs)	Mean ± std	Avg timeout rate	Engine freezes
vLLM v0.25.1, --gdn-prefill-backend triton	61.80, 55.06, 56.18, 56.18, 57.30	57.3 ± 2.6	~13%	0
SGLang v0.5.15.post1 (defaults + qwen3 reasoning parser)	56.18, 56.18, 60.67, 51.69, 56.18	56.2 ± 3.2	~17%	0

• Scaling 2→4 replicas doubled fast-phase throughput (~50% more tokens/episode) but did not move pass@1 or the timeout rate: the benchmark's deep tail is bounded by the per-task wall-clock budget, not aggregate serving capacity.
• Both stacks ran ~20h each under full-benchmark load with zero engine freezes — SGLang out of the box (after a readiness-probe adjustment: its /health generates a token and can exceed a 1s probe timeout, flapping healthy pods out of the Service; set probe timeoutSeconds ≥ 15), vLLM with the Triton GDN kernel noted above. Scores are statistically indistinguishable; vLLM sustained ~40% more tokens/episode at lower timeout rates on this workload.

Harness validation: gold-patch (oracle) run — 89/89

Ran all 89 TB 2.1 tasks with Harbor's OracleAgent, which uploads each task's reference solution/ and runs it instead of generating with a model. This exercises the sandbox backend, upload_dir, exec and the verifier end-to-end with the model removed from the equation, so the expected score is a clean sweep.

Result: 89/89 (reward 1.0 on every task), 22 min wall clock, zero agent timeouts.

Getting there surfaced one real bug in this integration, now fixed in this PR:

default_exec_timeout_s was silently truncating verification. Harbor's verifier calls environment.exec() without timeout_sec, so this default — not the task's declared [verifier] timeout_sec — is what bounds verification. The old values (300 in the environment, 600 in the TB config) were justified by reasoning only about Terminus-2's short tmux plumbing, which missed the verifier entirely. 87 of the 89 TB 2.1 tasks declare a verifier budget above 600s (median 900, max 12000), so verification was being killed mid-run and the task scored 0 instead of failing loudly.

exec cap	gold-patch score	what changed
600s	83/89	compile-compcert killed at 602s; query-optimize verifier killed at 604s → RewardFileNotFoundError
1200s	83/89	query-optimize fixed; compile-compcert still killed
1800s	85/89	both fixed
12000s	85/89	no further gain

Longest single command observed was 1384s, so 1800s is the smallest value that costs nothing — now the default in both the environment and this config. Because this bug suppressed slow-verifying tasks, previously reported pass@1 numbers are conservative.

The remaining 4 gold-patch failures were not harness issues and are deliberately not fixed here — three were upstream drift (a dataset host that refuses this network's egress; RcppParallel 6.0.0 requiring cmake, published 4 days before the run; planarity 1.0.0 renaming attributes pyknotid reads, published 4 weeks before) and one was macOS ._* sidecar files in our local dataset copy. Patching those reference solutions would only change an oracle run — solution/ is never uploaded for a model agent — so it cannot affect a scored eval, and fixing them for model runs would mean altering the task environment rather than the solution. Worth noting for anyone reproducing: sandbox resources were not implicated — the config's uniform 8 CPU / 16 GiB / 30 GiB is a strict superset of what all 89 task.toml files declare (max 4 CPU / 8 GiB / 10 GiB).

Final: card-parity cohort (prefix cache ON + Triton GDN kernel, 3h agent timeout — model-card conditions)

Run	pass@1	agent-timeout rate
1	60.67% (54/89)	10.1%
2	61.80% (55/89)	9.0%
3	57.30% (51/89)	13.5%
4	66.29% (59/89)	11.2%
5	59.55% (53/89)	13.5%
Mean	61.1 ± 3.3 (model card: 59.3)	11.5%

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.