feat: Implement Gym agent server for Codex CLI agent harness.

[ffrujeri]

Summary

Integrates the OpenAI Codex CLI (codex exec) as a built-in NeMo Gym agent harness, mirroring the Claude Code agent's architecture (#1336, #1603). Any Gym environment can now evaluate models inside the Codex harness — against OpenAI directly, or against any backend Gym serves via a model server. Implements #1389 (part of #1042).

The central constraint: Codex removed wire_api = "chat" (openai/codex#7782) and only speaks the Responses API over SSE. Where Claude Code needed a Messages ↔ Responses mapping (/v1/messages), Codex needs Gym's native format plus a streaming envelope and tolerance for its wire dialect. So this PR has two parts:

Core seam — every Gym model server speaks the streaming Responses dialect (nemo_gym/responses_streaming.py, dispatch route on SimpleResponsesAPIModel):

• Non-streaming /v1/responses requests are unchanged (strict validation, same 422 shape).
• stream: true requests are sanitized onto the strict params model and the complete response is re-emitted as synthesized SSE (response.created → response.output_item.done per item → response.completed — the minimal sequence streaming clients require, verified against codex-cli 0.144.4).
• The sanitizer degrades gracefully instead of 422-ing the rollout: drops unknown bookkeeping fields (client_metadata, prompt_cache_key), flattens namespace tool specs to <ns>__<tool> functions (and splits the names back into namespace + name on the way out — the wire shape Codex's tool router requires), hoists function tools out of code-mode additional_tools carriers, hoists leading developer messages into instructions (strict chat backends admit exactly one system message at position 0), and prunes SDK-unknown nested fields via generic extra_forbidden retry (e.g. reasoning.context).
• Converter fixes in responses_converter.py: instructions was silently dropped in Responses → ChatCompletions conversion — now mapped to a leading system message (also fixes the latent loss of Claude Code's system prompt on the vLLM path); when instructions is present, the leading system/developer run is folded into that single system message.

Agent — responses_api_agents/codex_agent/:

• Runs codex exec --json --ephemeral --skip-git-repo-check per request with a fresh generated CODEX_HOME (config.toml) and scratch --cd workdir, both removed afterwards.
• Gym-owned model provider with env_key auth — no codex login/auth.json needed. model_server ref (with /ng-rollout/<id> capture correlation) or direct openai_base_url.
• Isolation defaults: approval_policy=never, analytics/history/update-checks off, web_search disabled, features.multi_agent/code_mode off. sandbox_mode configurable (default danger-full-access, mirroring the Claude agent's skip-permissions default — Gym envs provide their own isolation).
• Model-name pinning: for models Codex recognizes, model-family gating enables code mode, which moves tools into carriers chat-served models cannot drive; model: null + model_server writes a gym-policy-model placeholder to keep classic function tools (Gym servers substitute their configured model anyway).
• Gym MCP tools: /seed_session metadata becomes a per-rollout streamable-HTTP mcp_servers entry with the session token on http_headers.
• Skills staged into CODEX_HOME/skills/ from the run-level skills_ref (native discovery, no --bare analog needed).
• Timeout is the runaway bound (no --max-turns equivalent): kill on timeout → empty response → verify still runs (reward 0). Codex runs in its own process group and timeout kills the whole group — the npm shim's vendored-binary child otherwise survives and holds the stdout pipe open past the kill.
• JSONL events map to a uniform trajectory shape (agent_message/reasoning → messages with <think> prefixes; command_execution/mcp_tool_call/file_change/web_search/todo_list → function_call + function_call_output pairs; turn.completed → usage incl. cached/reasoning details). A run ending on buffered reasoning is surfaced as a think-tagged message (vLLM reasoning parsers sometimes route the final answer there).
• extra_config deep-merge knob for arbitrary config.toml content; per-rollout Gym MCP entries win name collisions.

Also: composed showcase configs reasoning_gym_codex_agent[_model_server].yaml, README(s), agent-skills docs updated with the Codex staging location.

Test plan

Unit tests pass (full suite 1270 passed, 31 skipped; ruff + pre-commit clean). New coverage: ~45 agent tests (command/config/TOML generation, CODEX_HOME staging + cleanup, timeout/process-group kill, MCP config, rollout correlation, skills forwarding, event parsing) and ~28 core tests (sanitizer, SSE synthesis, dispatch route incl. non-streaming strict-validation regression, converter instructions mapping).

Unit tests mock asyncio.create_subprocess_exec; the real CLI and full stack were smoke-tested against codex-cli 0.144.4 with a vLLM-dialect model server (inference-api Qwen3.6-35B backend):

• Streaming dialect via curl: stream: true + codex bookkeeping fields against a live vllm_model server returns response.created / output_item.done / response.completed SSE.
• Real CLI shell round trip: codex exec against the Gym model server — model called exec_command, Codex executed /bin/bash -lc 'echo $((17*23))' → 391, multi-turn replay (reasoning + function_call + output) validated, turn.completed usage reported.
• MCP round trip: header-gated streamable-HTTP FastMCP server (the Gym resources-server stack) — namespaced function_call routed to the MCP tool with the per-rollout session token, result returned to the model.
• Full eval loop: gym env start --resources-server reasoning_gym/reasoning_gym_codex_agent_model_server --model-type vllm_model + gym eval run --no-serve ... --limit 2 → 2/2 rollouts with rewards, turns_used, token usage, and aggregate metrics.
• Timeout degradation: one live rollout exceeded the 600 s budget and landed as an empty, reward-0 rollout (and exposed the orphaned-child kill bug this PR fixes with process-group kill).

Smoke test details (reproducible)

env.yaml (any OpenAI-compatible chat endpoint):

policy_base_url: https://<host>/v1
policy_api_key: <key>
policy_model_name: <model>

gym env start \
  --resources-server reasoning_gym/reasoning_gym_codex_agent_model_server \
  --model-type vllm_model

gym eval run --no-serve \
  --agent reasoning_gym_codex_agent_model_server \
  --input resources_servers/reasoning_gym/data/example.jsonl \
  --output codex_rollout.jsonl --limit 1

Standalone CLI seam check (no resources server) — take the model server URL from the gym env start log:

mkdir -p /tmp/codex_home && cat > /tmp/codex_home/config.toml <<EOF
model = "<policy_model_name>"   # explicit name avoids model-family code-mode gating
model_provider = "gym"
approval_policy = "never"
sandbox_mode = "danger-full-access"
[model_providers.gym]
name = "gym"
base_url = "$URL/v1"
env_key = "GYM_API_KEY"
wire_api = "responses"
EOF
CODEX_HOME=/tmp/codex_home GYM_API_KEY=local \
  codex exec --json --ephemeral --skip-git-repo-check "Compute 17*23 with a shell command." < /dev/null

Limitations

• Eval only — token IDs / logprobs not wired up (same as the Claude Code agent).
• Token counts come from Codex's own turn.completed usage reporting; turns_used counts assistant messages.
• Code mode (JS tool orchestration) is fundamentally incompatible with chat-served backends; the seam degrades it gracefully (function tools preserved, JS exec dropped with a warning), but the supported path is classic function tools via explicit model naming.
• reasoning_gym_codex_agent* configs are verified: false pending reward-profile baselining.

Closes #1389.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-ffrujeri-feat-codex-agent.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/agent-server/agent-skills

[ananthsub]

LGTM! a few small comments inline, but overall this is great!