Fix prompt template path listing when CWD differs from project root

[simonrosenberg]

Summary

• Fix ValueError in prompt template choice listing when the current working directory is not the project root
• Changed p.relative_to(Path.cwd()) → str(p) in all 6 run_infer.py files
• The prompt directory is already resolved to an absolute path, so str(p) gives a reliable absolute path regardless of CWD

Files changed

File	Change
benchmarks/commit0/run_infer.py	p.relative_to(Path.cwd()) → str(p)
benchmarks/multiswebench/run_infer.py	p.relative_to(Path.cwd()) → str(p)
benchmarks/swebench/run_infer.py	p.relative_to(Path.cwd()) → str(p)
benchmarks/swebenchmultimodal/run_infer.py	p.relative_to(Path.cwd()) → str(p)
benchmarks/swefficiency/run_infer.py	p.relative_to(Path.cwd()) → str(p)
benchmarks/swtbench/run_infer.py	p.relative_to(Path.cwd()) → str(p)

Context

Extracted from #455. The old code used p.relative_to(Path.cwd()) to build the --prompt-template argparse choices list. This raises ValueError when CWD is not an ancestor of the prompt directory (e.g., when running benchmarks as an installed package or from a different directory).

Validation

All benchmarks pass with this change — see CI results on #455:

Benchmark	Status	Run
swebench	✅	run
swtbench	✅	run
commit0	✅	run
swebenchmultimodal	✅	run
gaia	✅	run

Test plan

• Verify benchmarks can be launched from a directory other than the project root
• Verify --prompt-template argument still works with the listed choices

🤖 Generated with Claude Code

[all-hands-bot]

🟡 Acceptable - Fixes a real bug but there's a cleaner solution.

VERDICT: ✅ Worth merging - Core logic is sound, improvement suggested

KEY INSIGHT: You're solving the right problem (CWD fragility) but using absolute paths in help text degrades UX when a simpler solution exists.

[all-hands-bot]

🟢 Code Smell: This identical change in 6 files suggests you could extract get_prompt_choices(prompt_dir) to a shared utility in benchmarks/utils/. Not critical, but reduces duplication and makes future fixes easier.

[all-hands-bot]

Additional observation on code structure:

[all-hands-bot]

🟢 Nit - Code Duplication: This exact pattern (prompt directory setup + choices list + argparse) appears identically in all 6 benchmark files. Consider extracting to benchmarks/utils/prompts.py:

def get_prompt_arg_parser(script_file: Path) -> tuple[list[str], Path]:
    """Get prompt choices and default path for argparse."""
    prompt_dir = (script_file.parent / "prompts").resolve()
    choices = [p.name for p in prompt_dir.glob("*.j2")]
    default = prompt_dir / "default.j2"
    assert default.exists(), f"Default prompt {default} not found"
    return choices, default

Not critical for this PR, but would make the next fix easier (you'd update one place instead of six).