Skip to content

feat(m3): prime multi-turn dialogue history so the last turn is a real follow-up - #176

Draft
haroldship wants to merge 2 commits into
feat/125-capability4-dataset-cleanupfrom
feat/m3-multiturn-dialogue-priming
Draft

haroldship wants to merge 2 commits into
feat/125-capability4-dataset-cleanupfrom
feat/m3-multiturn-dialogue-priming

Conversation

@haroldship

Copy link
Copy Markdown
Collaborator

Summary

Split out of PR #127 — this wasn't part of issue #125's ask, it's a new
agent-behavior change discovered while integrating the capability_4
dataset. Stacked on #175 (targets that branch, not main) because it
restructures the same function #175's policy-text wiring (item 5) touches —
review/merge #175 first.

Problem

VAKRA's capability_4 dialogue samples with 2+ turns hold every turn except
the last one already answered (each prior turn carries a gold "answer").
The eval loop live-invoked the agent on every turn in sequence, so the
agent had to actually re-solve turns whose answers were already known —
never a genuine follow-up question, always a fresh multi-step task.

Changes

  • sdk_eval_helpers.py: optional history_messages param on
    evaluate_task_with_langfuse; when given, prepended to the live turn's
    message so agent.invoke() receives the whole primed conversation in one
    call (the agent only ever answers the last message — everything before it
    is context, not re-solved).
  • eval_m3.py: evaluate_multiturn_task splits 2+-turn samples into prior
    turns (query + gold answer, stringified into synthetic
    HumanMessage/AIMessage history) and the final live turn; only the live
    turn is invoked and scored.
  • eval_m3.py: cuga-agent's ## User Context injection only fires on a
    thread's very first message — a primed thread never satisfies that, so
    policy text would silently never reach the model. Embedded directly on
    the live turn's message instead.
  • eval_m3.py: expected_output now reads gold_sequence/
    answer_per_turn/tool_response_per_turn from index -1 (the live/last
    turn) instead of 0 — equivalent for single-turn samples, correct for
    multi-turn.
  • CodeRabbit review fixups: GenericReactAgent fails loudly instead of
    silently dropping primed history (not currently reachable — M3's
    --agent react never calls this path — but a shared helper other
    benchmarks call); guard gt_by_turn construction against malformed
    ground-truth entries.

Tests

uv run --no-sync pytest benchmarks/m3/tests/ benchmarks/helpers/tests/ -m sanity:
180 passed, 2 skipped (pre-existing/unrelated).

Test plan for reviewers

…l follow-up

VAKRA's capability_4 dialogue samples with 2+ turns hold every turn except
the last one already answered (each prior turn carries a gold "answer").
Previously the eval loop live-invoked the agent on every turn in sequence,
so the agent had to actually re-solve turns whose answers were already
known - never a genuine follow-up question, always a fresh multi-step task.

- sdk_eval_helpers.py: add optional history_messages param to
  evaluate_task_with_langfuse; when given, prepended to the live turn's
  message so agent.invoke() receives the whole primed conversation in one
  call (the agent only ever answers the last message - everything before
  it is context, not re-solved)
- eval_m3.py: evaluate_multiturn_task splits samples with 2+ turns into
  prior turns (query + gold answer, stringified into synthetic
  HumanMessage/AIMessage history) and the final live turn; only the live
  turn is invoked and scored
- eval_m3.py: cuga-agent's "## User Context" injection (the mechanism
  additional_instructions/user_context rides on) only fires on a thread's
  very first message - a primed thread never satisfies that, so it would
  silently never reach the model. Embed it directly on the live turn's
  message instead, in the same format, so the policy still applies
- eval_m3.py: expected_output now reads gold_sequence/answer_per_turn/
  tool_response_per_turn from index -1 (the live/last turn) instead of 0
  (the first turn) - equivalent for single-turn samples, correct for
  multi-turn; _annotate_tool_call_diffs only compares the live turn since
  primed turns were never actually invoked

Part of #125
- sdk_eval_helpers.py: history_messages (dialogue priming) silently
  dropped every primed turn when the agent is GenericReactAgent - its
  invoke() only ever reads messages[-1], building one templated prompt
  rather than a message-list conversation like CugaAgent. Not currently
  reachable (M3's --agent react routes through the separate
  eval_m3_react.py, which never calls evaluate_multiturn_task), but this
  is a shared helper other benchmarks also call. Fail loudly instead of
  silently dropping context if this combination is ever hit.
- m3_data_loader.py: guard gt_by_turn construction against non-dict
  entries in gt_list, so a malformed/null ground-truth entry raises a
  clear KeyError-free skip instead of AttributeError on .get()

Part of #125
@coderabbitai

coderabbitai Bot commented Aug 26, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 1c25d0fc-3967-4b93-b0e7-224a7145f745

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant