Model routing: route Safe Output Health Monitor instead of defaulting to claude-sonnet-5 #73027

Summary

Route Safe Output Health Monitor with model routing, using goal: cost and mode: economy, instead of running every day on the Claude engine's default model, claude-sonnet-5. Only the frontmatter of .github/workflows/safe-output-health.md and the recompiled safe-output-health.lock.yml change. The agent then reaches Claude through GitHub Copilot instead of the Anthropic API key. Estimated saving: about 2,600 AI Credits (AIC; 1 AIC = $0.01) a week.

This is one of four issues filed together for gh-aw's daily Claude Code report workflows. The others are Copilot Session Insights (#73024), Claude Code User Documentation Review (#73025) and Agent Job Health Monitor (#73026).

Why model routing

Safe Output Health Monitor runs every task on one model, however easy. Model routing picks the model and reasoning effort for each run, based on its task, so the workflow no longer depends on one model name staying available. We're moving gh-aw's own workflows over one at a time, one per PR, and measuring cost and outcomes after each.

The first, Duplicate Code Detector, has been routed since 17 October (#69499):

With so few runs, we read the quality result as "no worse", not "better". Avenger is next (#73009).

With the Claude engine, model routing runs through GitHub Copilot (model-provider: github) and chooses only among the claude-* models that Copilot offers. smoke-claude-routed already uses this engine configuration, with its candidates limited to claude-sonnet-5. Its last 12 runs, 18 to 29 October, all succeeded.

Why Safe Output Health Monitor next

• Almost all of its cost is the agent on claude-sonnet-5. Safe Output Health Monitor cost 2,419, 1,460, 3,210 and 2,466 AIC in the weeks starting 28 September, 5, 12 and 19 October. A run whose agent executes costs about 406 AIC: about 405 for the agent and about 1 for threat detection, which already use their default models. At its recent run rate, that's about 2,840 AIC a week.
• It reads data and writes a report. It analyzes safe-output job results across all agentic workflows and posts a health report as a discussion. We expect the router to classify a run as a medium-difficulty explanation within one subsystem.
• A much cheaper choice scores about the same. For that task, with goal: cost and mode: economy, the routing tables in AWF v0.29.0, gh-aw's current default, pick claude-haiku-5.5 at xhigh effort. On the router's benchmarks it scores 40.7, against about 37 for claude-sonnet-5 across its efforts. It costs about 8 AIC per benchmark task, against 85–107 for claude-sonnet-5 at medium and high. The benchmarks are coding tasks, so they're weaker evidence for a report than for a code fix. We estimate routed runs at about 35 AIC each, about 250 AIC a week.
• A weaker run does little harm. Its only output is one discussion in the audits category (an issue, if the discussion can't be created). 7 of its last 10 runs, 20 to 29 October, succeeded. Two of the failures were in the conclusion job only; on 25 October the agent failed too. That's the baseline to watch.

Why economy. Safe Output Health Monitor reads data and reports on it, and people read the report before acting on it. economy already picks a model that scores above claude-sonnet-5 on the router's benchmarks. If reports get worse, balanced picks claude-opus-5.5 at low effort. That scores 48.4 at about 31 AIC per benchmark task, still well below claude-sonnet-5.

Why not auto. Safe Output Health Monitor's prompt is the same every day; only the data it reads changes. The docs recommend a fixed mode when you know what kind of tasks a workflow gets.

Proposed change

In .github/workflows/safe-output-health.md:

• Keep engine.id: claude and engine.mcp.tool-timeout: 10m.
• Add engine.model-provider: github. The Claude engine needs it for model routing.
• Add engine.model-routing with goal: cost and mode: economy.

The resulting engine block:

engine:
  id: claude
  model-provider: github
  model-routing:
    goal: cost
    mode: economy
  mcp:
    tool-timeout: 10m

Then run make recompile. Only safe-output-health.md and safe-output-health.lock.yml should change. In the lock file, expect these changes:

• The AWF router image is added to the pinned images.
• A "Prepare model-routing conversation" step is added.
• Model routing is enabled in the agent's AWF config, with github-copilot/claude-* as the candidates.
• The agent no longer uses ANTHROPIC_API_KEY. The secret, its validation step and the GH_AW_MODEL_AGENT_CLAUDE default are removed. Claude Code reaches Copilot through the AWF API proxy, authenticated with the COPILOT_GITHUB_TOKEN secret.
• The routed model, effort and routing status are passed to the downstream jobs.
• Threat detection runs on Copilot CLI with its detection default model, instead of on Claude Code.

On current main (e6afe3424d), this change compiles to exactly that diff.

The compiler's warning about mcp__agenticworkflows__status is already there on main and is unrelated.

The docs note that Claude Code fails at none or minimal effort. The current routing tables have no such choices for Claude models, so the router can't pick them.

After merge

Over the first two weeks:

• Check the first routed runs with gh aw audit <run>. Its Model Routing section shows the selected model and effort, as do the firewall.model_routing records in usage/aw_session.jsonl. Expect mostly claude-haiku-5.5 at xhigh. The router classifies each run, so some runs can differ.
• Compare AIC per run, for the agent and the threat detection job together, with about 406 AIC today.
• Compare the success rate with 7 of its last 10 runs, and read the first few reports against recent ones. If the reports get worse, switch to mode: balanced. To roll back, remove model-provider and model-routing and recompile.

Merge order. We suggest merging Copilot Session Insights (#73024) and Claude Code User Documentation Review (#73025) first. Merge the two health monitors, Agent Job Health Monitor (#73026) and Safe Output Health Monitor (#73027), after the first routed runs of those two look right, on different days, so each change can be measured on its own.

Out of scope

• Other workflows. Each gets its own issue.
• Safe Output Health Monitor's prompt, schedule, tools, permissions, limits and safe outputs.
• Changes to the router or its tables.

Acceptance criteria

• safe-output-health.md sets engine.model-provider: github and engine.model-routing with goal: cost and mode: economy, and keeps engine.id: claude and engine.mcp.tool-timeout: 10m.
• safe-output-health.lock.yml is regenerated with make recompile, and no other workflow changes.