feature: shadow a request against a second model to measure where the cheap model is good enough

[mrsabath]

Feature Description

#1019 and #1020 both rest on an unproven assumption: that for some
classes of request, a cheaper model is as good as an expensive one. Nobody on the team has data
for that on our own workloads.

This issue proposes measuring it. In an opt-in mode, Cortex sends a request to two models —
the one the agent asked for and a cheaper candidate — returns the primary's response to the
agent unchanged, and compares the two offline. Over a working week that produces a
per-task-class answer to "could we have used the cheap model here", grounded in the team's real
traffic rather than a public benchmark.

Proposed Solution

• Shadow mode: dispatch to a secondary model in parallel; the agent only ever receives the
  primary response, and shadow latency never delays it
• Comparison, tiered by how much it costs to run:
  • exact / normalised match — cheap, and the only meaningful check for structured output like
    tool calls and JSON
  • agreement on the decision rather than the prose — did both pick the same tool with the
    same arguments
  • semantic similarity, and LLM-as-judge as an explicitly-priced opt-in tier
• Record per pair: agreement rate, token and cost delta, latency delta, broken down by task
  class — the classes feature: route by task class — use a cheaper model when the prompt does not need an expensive one #1020 wants to route on
• Report the cost of the experiment itself honestly; shadow mode roughly doubles spend
  while enabled, and that must be visible before a user turns it on
• Hard off by default, with a bounded budget or a request cap, so nobody leaves it running
  by accident

On benchmarks

Two separable things, and the second is more valuable to us:

• Validate against existing public benchmarks — useful sanity check, but public benchmarks
  do not resemble our agent traffic, and the frontier models are trained near them
• Build a Cortex-native benchmark from captured real traffic — a corpus of the team's own
  requests with recorded model outputs, replayable against any candidate model. This is the thing
  that would actually justify a routing decision, and it depends on feature: persist sessions #901 (session persistence)
  for the capture layer

Captured traffic is sensitive: any such corpus needs a redaction and consent story before it
exists, not after. Worth scoping that as its own issue if this one is taken up.

Risks and open questions

• "Same result" is doing a lot of work. Two prose answers are almost never identical and
  often equally good; two tool calls either match or don't. The tool-call and structured-output
  case is where this method is sound — start there and be honest that free-text comparison is
  weaker
• Doubling requests changes rate-limit behaviour and may trip the very limits feature: transparent model and provider switching — continue a session across a limit, price or policy boundary #1019
  exists to handle
• Cost of measurement vs value of the answer: this could easily cost more than it saves if left
  on indefinitely. It is a time-boxed experiment feature, not a steady-state one

Additional Context

Team brainstorming, September 2026 — cc @aslom and @galmasi. Post-v0.9.0. Supplies the evidence base for #1020 and shares
the provider abstraction from #1019. Depends on #901 for capture and #950/#951 for the cost and latency numbers it
reports.

Part of the post-v0.9.0 brainstorm backlog under #962.