feat(m3): prime multi-turn dialogue history so the last turn is a real follow-up - #176
Draft
haroldship wants to merge 2 commits into
Draft
haroldship wants to merge 2 commits into
haroldship wants to merge 2 commits into
Conversation
…l follow-up VAKRA's capability_4 dialogue samples with 2+ turns hold every turn except the last one already answered (each prior turn carries a gold "answer"). Previously the eval loop live-invoked the agent on every turn in sequence, so the agent had to actually re-solve turns whose answers were already known - never a genuine follow-up question, always a fresh multi-step task. - sdk_eval_helpers.py: add optional history_messages param to evaluate_task_with_langfuse; when given, prepended to the live turn's message so agent.invoke() receives the whole primed conversation in one call (the agent only ever answers the last message - everything before it is context, not re-solved) - eval_m3.py: evaluate_multiturn_task splits samples with 2+ turns into prior turns (query + gold answer, stringified into synthetic HumanMessage/AIMessage history) and the final live turn; only the live turn is invoked and scored - eval_m3.py: cuga-agent's "## User Context" injection (the mechanism additional_instructions/user_context rides on) only fires on a thread's very first message - a primed thread never satisfies that, so it would silently never reach the model. Embed it directly on the live turn's message instead, in the same format, so the policy still applies - eval_m3.py: expected_output now reads gold_sequence/answer_per_turn/ tool_response_per_turn from index -1 (the live/last turn) instead of 0 (the first turn) - equivalent for single-turn samples, correct for multi-turn; _annotate_tool_call_diffs only compares the live turn since primed turns were never actually invoked Part of #125
- sdk_eval_helpers.py: history_messages (dialogue priming) silently dropped every primed turn when the agent is GenericReactAgent - its invoke() only ever reads messages[-1], building one templated prompt rather than a message-list conversation like CugaAgent. Not currently reachable (M3's --agent react routes through the separate eval_m3_react.py, which never calls evaluate_multiturn_task), but this is a shared helper other benchmarks also call. Fail loudly instead of silently dropping context if this combination is ever hit. - m3_data_loader.py: guard gt_by_turn construction against non-dict entries in gt_list, so a malformed/null ground-truth entry raises a clear KeyError-free skip instead of AttributeError on .get() Part of #125
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
This was referenced Aug 26, 2026
haroldship
marked this pull request as draft
August 27, 2026 14:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Split out of PR #127 — this wasn't part of issue #125's ask, it's a new
agent-behavior change discovered while integrating the capability_4
dataset. Stacked on #175 (targets that branch, not
main) because itrestructures the same function #175's policy-text wiring (item 5) touches —
review/merge #175 first.
Problem
VAKRA's capability_4 dialogue samples with 2+ turns hold every turn except
the last one already answered (each prior turn carries a gold "answer").
The eval loop live-invoked the agent on every turn in sequence, so the
agent had to actually re-solve turns whose answers were already known —
never a genuine follow-up question, always a fresh multi-step task.
Changes
sdk_eval_helpers.py: optionalhistory_messagesparam onevaluate_task_with_langfuse; when given, prepended to the live turn'smessage so
agent.invoke()receives the whole primed conversation in onecall (the agent only ever answers the last message — everything before it
is context, not re-solved).
eval_m3.py:evaluate_multiturn_tasksplits 2+-turn samples into priorturns (query + gold answer, stringified into synthetic
HumanMessage/AIMessage history) and the final live turn; only the live
turn is invoked and scored.
eval_m3.py: cuga-agent's## User Contextinjection only fires on athread's very first message — a primed thread never satisfies that, so
policy text would silently never reach the model. Embedded directly on
the live turn's message instead.
eval_m3.py:expected_outputnow readsgold_sequence/answer_per_turn/tool_response_per_turnfrom index -1 (the live/lastturn) instead of 0 — equivalent for single-turn samples, correct for
multi-turn.
GenericReactAgentfails loudly instead ofsilently dropping primed history (not currently reachable — M3's
--agent reactnever calls this path — but a shared helper otherbenchmarks call); guard
gt_by_turnconstruction against malformedground-truth entries.
Tests
uv run --no-sync pytest benchmarks/m3/tests/ benchmarks/helpers/tests/ -m sanity:180 passed, 2 skipped (pre-existing/unrelated).
Test plan for reviewers
uv run --no-sync pytest benchmarks/m3/tests/ benchmarks/helpers/tests/ -m sanity