NeMo Evaluator Integration

[simonrosenberg]

Summary

Adds NVIDIA NeMo Evaluator integration so OpenHands benchmarks can be run via NeMo's evaluation framework. The new nemo_evaluator package is an optional dependency (pip install .[nemo]) that bridges NeMo to the existing benchmark CLI entrypoints.

Note: Several cross-cutting refactors that were originally part of this PR have been extracted and already merged separately:

• Rename image_exists → remote_image_exists (Rename image_exists to remote_image_exists #471)
• Replace uv run python with sys.executable in eval scripts (Replace uv run python with sys.executable in eval scripts #472)
• Centralize IMAGE_TAG_PREFIX for Docker image tags (Centralize IMAGE_TAG_PREFIX for Docker image tags #473)

New: nemo_evaluator/openhands_benchmarks/ package

File	Purpose
framework.yml	NeMo framework definition with 16 named evaluation configs (SWE-bench variants, GAIA, Commit0, Multi-SWE-bench per language, SWT-bench, SWE-bench Multimodal, OpenAgentSafety)
run_benchmark.py	Orchestrator that maps NeMo parameters to existing benchmark CLI entrypoints, runs inference then evaluation
generate_llm_config.py	Generates LLM config JSON from CLI args (resolves API key env vars, strips /chat/completions suffix for LiteLLM)
output.py	Parses *.report.json output into NeMo's EvaluationResult format (accuracy = resolved/submitted)

New console script entrypoints: generate-llm-config, run-benchmark.

pyproject.toml

• Optional dependency: nemo = ["nemo_evaluator>=0.1.92,<0.2"]
• New entrypoints: generate-llm-config, run-benchmark
• Package discovery: ["benchmarks*", "nemo_evaluator*"]
• Package data: *.yml for nemo_evaluator, *.j2/Dockerfile*/*.json for benchmarks
• Removed exclude = ["legacy"] from ruff config (moved to .pre-commit-config.yaml exclude)

.pre-commit-config.yaml

• Added nemo_evaluator/ to the exclude pattern (NeMo-specific code uses its own conventions)

Previous PRs

Continues from #412 (closed). Refactors extracted into separate PRs:

• image_exists → remote_image_exists rename (Rename image_exists to remote_image_exists #471)
• uv run python → sys.executable (Replace uv run python with sys.executable in eval scripts #472)
• Centralized IMAGE_TAG_PREFIX (Centralize IMAGE_TAG_PREFIX for Docker image tags #473)
• load_llm_config shared utility (Extract shared load_llm_config utility #453)
• Multi-SWE-Bench dataset name check (Fix Multi-SWE-Bench dataset name check #454)

[all-hands-bot]

Testing Gap: The new nemo_evaluator package has no test coverage.

[all-hands-bot]

🟠 Important: No tests for new integration package.

You've added 3 new modules (generate_llm_config.py, run_benchmark.py, output.py) totaling ~400 lines with zero test coverage. This is an integration shim to NeMo - if it breaks, debugging will be painful.

Minimum viable tests:

1. test_generate_config() - verify JSON output format, env var resolution, URL stripping
2. test_build_infer_cmd() - verify command building for each benchmark type
3. test_parse_output() - verify report.json parsing and accuracy calculation

"Testing and practice sometimes clash. Testing wins. Every single time." - If this breaks in production, you'll wish you had tests to reproduce the issue locally.

Add tests in nemo_evaluator/tests/ before merging.

[simonrosenberg]

Nemo Evaluator Integration Validation Results

Triggered CI benchmarks with eval_limit=1 to validate the Nemo Evaluator integration.

Results: 5/6 Benchmarks Passed ✅

Benchmark	Status	Evaluation Run
gaia	✅ Success	https://github.com/OpenHands/evaluation/actions/runs/22590769837
swebench	✅ Success	https://github.com/OpenHands/evaluation/actions/runs/22590769394
swtbench	✅ Success	https://github.com/OpenHands/evaluation/actions/runs/22590775404
commit0	✅ Success	https://github.com/OpenHands/evaluation/actions/runs/22590778789
swebenchmultimodal	✅ Success	https://github.com/OpenHands/evaluation/actions/runs/22590785406
multiswebench	❌ Failed	Dataset schema issue (unrelated to this PR)

Configuration

• SDK Branch: main (c31cdee)
• Model: claude-sonnet-4-5-20250929
• Eval Limit: 1 instance per benchmark

Note on multiswebench failure

The multiswebench failure is due to a known HuggingFace dataset schema incompatibility with bytedance-research/Multi-SWE-Bench - not related to this PR. Fix available in PR #304.

[juanmichelini]

Results: 5/6 Benchmarks Passed ✅

notice that for swebench, swtbench and commit0 the single instance was resolved.

[juanmichelini]

On second pass the evaluations were run with branch main and not branch nemo-evaluator

[simonrosenberg]

On second pass the evaluations were run with branch main and not branch nemo-evaluator

damn it

[juanmichelini]

@OpenHands see software-agent-sdk, run 5 eval job actions there, one for each of these benchmarks: commit0, gaia, swebench, swebenchmultimodal, gaia. for each of those run a single instance, in all use trigger reason "testing benchmarks branch nemo-evaluator", make sure to set the benchmark branch param nemo-evaluator so that we can test that PR. Run it, the software agent sdk action will run an evaluation job. Once you have links to the 5 evaluation jobs post a comment here with the links.

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[juanmichelini]

Evaluation Jobs Triggered

Successfully triggered 5 evaluation jobs testing the nemo-evaluator branch with eval_limit=1:

Benchmark	Evaluation Job Link
commit0	https://github.com/OpenHands/evaluation/actions/runs/22594781791
gaia	https://github.com/OpenHands/evaluation/actions/runs/22594775753
swebench	https://github.com/OpenHands/evaluation/actions/runs/22594781250
swebenchmultimodal	https://github.com/OpenHands/evaluation/actions/runs/22594790024
swtbench	https://github.com/OpenHands/evaluation/actions/runs/22594786078

All jobs are configured with:

• SDK version: v1.11.5
• Benchmarks branch: nemo-evaluator
• Eval limit: 1 instance per benchmark
• Trigger reason: testing benchmarks branch nemo-evaluator

[openhands-ai]

Final Summary

I successfully completed your request to trigger evaluation jobs for testing the nemo-evaluator branch (PR #455).

Actions Completed:

✅ Triggered 5 evaluation job actions in the OpenHands/software-agent-sdk repository using the "Run Eval" workflow:

• commit0
• gaia
• swebench
• swebenchmultimodal
• swtbench

Each job was configured with:

• SDK version: v1.11.5
• Benchmarks branch: nemo-evaluator
• Eval limit: 1 instance per benchmark
• Trigger reason: "testing benchmarks branch nemo-evaluator"

✅ Posted a comment on PR #455 with links to all 5 evaluation jobs:

• commit0: https://github.com/OpenHands/evaluation/actions/runs/22594781791
• gaia: https://github.com/OpenHands/evaluation/actions/runs/22594775753
• swebench: https://github.com/OpenHands/evaluation/actions/runs/22594781250
• swebenchmultimodal: https://github.com/OpenHands/evaluation/actions/runs/22594790024
• swtbench: https://github.com/OpenHands/evaluation/actions/runs/22594786078

Comment URL: #455 (comment)

All evaluation jobs are now running and will test the NeMo Evaluator integration changes on the nemo-evaluator branch.

[all-hands-bot]

Taste Rating: 🟡 Acceptable - NeMo integration works but reveals abstraction issues.

Key Insight: Dict-of-lambdas returning tuple-lists with empty-string sentinel values is a code smell. This pattern works but signals the abstraction is leaking - as you add more benchmarks, maintenance cost grows non-linearly.

Critical Gap: Still no tests for 400+ lines of integration code (unresolved from previous review). This is NeMo glue - when it breaks, debugging will be painful without tests.

[all-hands-bot]

🔴 Critical: Excluding nemo_evaluator from pre-commit bypasses type checking and linting.

You've exempted this code from pyright and ruff. This means type errors and code quality issues will slip through. NeMo has its own conventions, but type safety and basic linting shouldn't be negotiable.

If NeMo's conventions conflict with your project's, document the specific conflicts and exclude only those rules. Blanket exclusion is how technical debt accumulates silently.

[all-hands-bot]

🟠 Important: Dict-of-lambdas is overengineered for simple conditional args.

This pattern:

BENCHMARK_INFER_PARAMS: dict[str, Callable[[...], list[tuple[str, str]]]] = {
    "gaia": lambda args: [("--level", args.level)] if args.level else [],
}

Could be:

def get_benchmark_args(benchmark: str, args: Namespace) -> list[str]:
    """Returns CLI args for benchmark-specific parameters."""
    if benchmark == "gaia" and args.level:
        return ["--level", args.level]
    elif benchmark == "commit0" and args.repo_split:
        return ["--repo-split", args.repo_split]
    # ...
    return []

Simpler data structures → simpler code. The dict-of-lambdas doesn't buy you anything except indirection.

[all-hands-bot]

🟡 Suggestion: Empty string as sentinel for bare flags is a code smell.

Using ("--modal", "") to mean "bare flag" requires this special-case check. Better approach: separate the flag building from the tuple structure, or use a simple boolean/None pattern.

Example:

for item in params:
    if isinstance(item, tuple):  # flag with value
        cmd.extend(item)
    else:  # bare flag
        cmd.append(item)

Or just handle it where the data is created instead of using empty strings.

[all-hands-bot]

🟠 Important: sorted()[-1] assumes lexicographic ordering gives "latest" - fragile.

If output_dir contains multiple runs with timestamps or run IDs in different formats, sorted() might not pick the latest. You're relying on filesystem/naming conventions that aren't enforced.

Better: Either error if len(output_files) != 1 (strict), or use file modification time:

output_jsonl = max(output_files, key=lambda p: p.stat().st_mtime)

Theory: "we'll only have one output.jsonl per run"
Practice: Multi-attempt runs, retries, or manual re-runs will break this assumption.

[all-hands-bot]

🟢 Good taste: Explicit check for multiple report files prevents ambiguity.

Unlike run_benchmark.py:247 which silently picks the last file, this correctly errors on ambiguous input (line 19-23). This is the right behavior for validation logic - fail fast and explicit rather than guessing.

[all-hands-bot]

🟡 Suggestion: Tight coupling via metadata file is pragmatic but document the contract.

output.py requires run_benchmark.py to write nemo_metadata.json. This works, but the coupling is implicit - nothing enforces that run_benchmark.py actually writes it, or what schema it uses.

Add a comment or type definition for the metadata schema:

# Expected schema: {"benchmark": str}
metadata = json.loads(metadata_file.read_text(encoding="utf-8"))

Or use a simple dataclass/TypedDict to document the contract.