Fix benchmark bugs across multiple benchmarks

[simonrosenberg]

Summary

Extracts the benchmark-only bug fixes from the NeMo Evaluator PR (#455) into a standalone PR. These are independent improvements to existing benchmarks that don't depend on or relate to NeMo integration.

Changes

Multi-SWE-Bench (multiswebench/)

• build_images.py: Use .get("version", "") instead of instance["version"] to avoid KeyError on instances missing the version field; call .lower() on Docker image names to avoid registry rejection of mixed-case tags; switch from get_dataset to download_and_concat_dataset to fix non-standard HuggingFace dataset structure; pass base_image_to_custom_tag_fn to build_all_images
• eval_infer.py: Re-raise errors from evaluation subprocess instead of silently swallowing them; wrap args.input_file in Path() to fix AttributeError when it's a plain string

OpenAgentSafety (openagentsafety/)

• build_images.py: Extract get_image_name() helper so the image name is configurable via EVAL_AGENT_SERVER_IMAGE and IMAGE_TAG_PREFIX env vars instead of being hardcoded; defer get_vendor_sdk_commit() until after the existing-image check to avoid unnecessary git calls
• run_infer.py: Wrap pd.isna() calls in try/except to prevent crashes on non-scalar types (dicts, lists); add Pydantic model_dump() support in NumpyEncoder; simplify NPC config writing (single heredoc command instead of tempfile + upload); add try/except fallback to pre-built image when on-the-fly build fails; use get_image_name() instead of hardcoded image strings; add max_retries pass-through to evaluator; add generate_report() to produce .report.json output

SWE-Bench (swebench/)

• eval_infer.py: Add default values from EVAL_DEFAULTS to run_swebench_evaluation() function parameters, enabling direct programmatic invocation without passing every argument; add explicit default for --timeout CLI flag

SWE-Bench Multimodal (swebenchmultimodal/)

• eval_infer.py: Document the modal parameter in run_swebench_multimodal_evaluation() docstring

SWT-Bench (swtbench/)

• run_infer.py: Lazy-import _base_slug from openhands.agent_server.docker.build to avoid git subprocess checks that fail when installed as a package outside a git repo

GAIA (gaia/)

• run_infer.py: Pass max_retries through to the evaluator (was silently ignored before)

Shared utilities (utils/)

• build_utils.py: Import TargetType from benchmarks.swebench.constants instead of openhands.agent_server.docker.build; lazy-import BuildOptions and build to avoid git subprocess failures when installed outside a git repo

Why these changes are necessary

These fixes address real bugs that cause crashes or incorrect behavior during benchmark evaluation:

1. Silent error swallowing in multi-swe-bench evaluation makes debugging impossible
2. KeyError crashes on missing version fields prevent building images for certain instances
3. Mixed-case Docker tags are rejected by container registries
4. pd.isna() on dicts/lists crashes the OpenAgentSafety JSON encoder
5. Hardcoded image names prevent using pre-built or custom images
6. Top-level git-dependent imports break when benchmarks are installed as a pip package
7. Missing max_retries pass-through means the retry config was silently ignored

Test plan

• Verify pre-commit checks pass
• Run SWE-bench eval with default args
• Run Multi-SWE-bench image build
• Run OpenAgentSafety inference

🤖 Generated with Claude Code

[all-hands-bot]

🔴 Critical regression found - This PR fixes real bugs but introduces a new crash. See inline comments for details.

[all-hands-bot]

Taste Rating: 🟢 Good taste - Pragmatic fixes for real bugs with proper tests

The critical unresolved_ids bug from the previous review is fixed, and the new tests actually exercise real code paths (not just mocks). Re-raising errors instead of swallowing them is exactly right. The lazy imports are justified workarounds with clear comments explaining why they're needed.

Verdict: ✅ Ship it

Key Insight: These are textbook pragmatic bug fixes - simple solutions that solve real crashes without over-engineering.