fix(m3): stop clobbering per-domain scoping and scope Vakra output files to the workspace bundle

[haroldship]

Summary

Split out of PR #121 into its own issue/PR: #169.

Fixes #169.

Bug 1: domains after the first silently lose all tools in --no-ground-truth mode

evaluate_single_task re-derived domains from
m3_data_loader.available_domains(task_id) and unconditionally overwrote
the caller's narrowed-down value with the full domain list for the task.
That discarded the single-domain narrowing that
rewrite_config_with_loader_domains + expand_registry_config already
apply upstream specifically so the sequential per-service loop can start one
registry per domain.

The first call (e.g. domain "address") completes fine; every domain after
it fails with Application 'airline' not found in registry / Loaded 0 tools for 'airline', because only the first domain's registry was ever
started. Observed on cap3-guard-test.

Bug 2: Vakra prediction/groundtruth files not scoped per experiment

evaluate_single_task wrote _vakra/prediction/<domain>.json (and, in GT
mode, _vakra/groundtruth/) to the shared benchmarks/m3/results/
directory regardless of experiment. Two concurrent eval.sh runs for
different experiments hitting the same domain name can clobber each other's
prediction file there, and old runs' files are never cleared.

Changes

1. Stop overwriting the caller's narrowed domains list.
2. Use bundle_dir/results when a workspace bundle exists (same fallback
   pattern _finalize_and_save_results already uses), so each named
   experiment gets its own isolated, self-cleaning directory tree. Legacy
   (un-experimented) runs keep the old shared path.

Tests

• test_sequential_domain_scoping.py: evaluate_single_task doesn't
  override domains from the loader; --no-ground-truth flag still computed
  correctly.
• test_vakra_output_scoping.py: predictions/groundtruth isolated by
  output dir; evaluate_single_task uses the bundle-scoped output dir when
  present.

Test plan for reviewers

• Run uv run --no-sync pytest benchmarks/m3/tests/test_sequential_domain_scoping.py benchmarks/m3/tests/test_vakra_output_scoping.py -v

[coderabbitai]

Warning

Review limit reached

• Run on-demand review

_{On-demand reviews are free for the next 24 days. After that, they cost $0.25 per reviewed file.}

Or wait 55 minutes for your next included review.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e47f54fd-cd12-4d44-926e-7349c81620dd

📥 Commits

Reviewing files that changed from the base of the PR and between 398acce and 3e15b12.

📒 Files selected for processing (3)

• benchmarks/m3/eval_m3.py
• benchmarks/m3/tests/test_sequential_domain_scoping.py
• benchmarks/m3/tests/test_vakra_output_scoping.py

---

_{Comment @coderabbitai help to get the list of available commands.}

[haroldship]

@coderabbitai full review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

[haroldship]

@coderabbitai full review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 6 minutes.

[haroldship]

@coderabbitai full review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 55 minutes.