Skip to content

buzz-agent: move proactive context handoff to end of turn - #6584

Open
tlongwell-block wants to merge 2 commits into
mainfrom
meli/end-of-turn-compaction
Open

buzz-agent: move proactive context handoff to end of turn#6584
tlongwell-block wants to merge 2 commits into
mainfrom
meli/end-of-turn-compaction

Conversation

@tlongwell-block

Copy link
Copy Markdown
Collaborator

Summary

Move buzz-agent's proactive context handoff from the start of every tool-loop round to once, at the end of the turn.

Problem. The round-start gate compacted while the turn's working set (file reads, tool results) was exactly what the model was still using. Summarizing it away mid-task forced a re-read storm that refilled the window. The gate also projected history growth at 1 byte/token, so a single large tool result could trip it on its own.

Change.

  • handoff.rs: maybe_handoff(&mut attempts)end_of_turn_handoff(). Gates on max_handoffs == 0 (disabled) and should_handoff(), which now compares the provider's measured input usage for the turn's final request directly against the 90% threshold. The growth estimator (projected_handoff_input_tokens) and its byte baseline (last_request_history_bytes) are deleted end-to-end. The pre-usage byte fallback is unchanged.
  • handoff() takes enum Reseat { LivePrompt, None }: end of turn does not re-append the already-answered prompt; the reactive context-400 path keeps LivePrompt because it continues the same turn.
  • agent.rs: per-turn handoff_attempts counter and round-start gate removed; truncate_history is unconditional after drain_steers(). Mid-turn overflow is handled only by the reactive context-400 ladder (unchanged: in-loop retry, round refund, independent 3-rung bound).
  • lib.rs: end_of_turn_handoff() runs after ctx.run() when the result is Ok and the stop reason is not Cancelledrun() reports cancellation as Ok(StopReason::Cancelled), so an is_ok() gate would spend a summarize the user just asked us to stop.
  • BUZZ_AGENT_MAX_HANDOFFS is now a switch: 0 disables proactive compaction, any positive value enables it (≤1 per turn by construction). Reactive recovery bypasses it as before. config.rs / README.md updated.

Deliberately not changed: the 90% threshold. Timing and threshold are not tuned in the same PR; measure first.

Related issue

None found. Supersedes the relay-hosted draft (buzz:// PR #1 by Pinky/Brain) that proposed the same move.

Testing

tests/regressions.rs rewritten for the new boundary (54 regressions, all green):

  • token_usage_over_budget_triggers_handoff — summarize lands inside the over-threshold turn; next request's user items are exactly [Context Handoff], new prompt (no re-seated answered prompt).
  • history_growth_mid_turn_does_not_trigger_handoff — 6 KB tool result mid-turn fires nothing; gate reads the final request's usage (8500+8900 cumulative > 9000 gate, final 8900 < 9000 → no handoff).
  • large_final_reply_does_not_trigger_end_of_turn_handoff — usage 8500 < 9000 with a 2 KB reply → exactly one request. Fails on the first commit, passes on the second (this was the review blocker: the old estimator charged the reply at 1 byte/token).
  • cancelled_turn_skips_end_of_turn_handoff — cancel during an in-flight provider call → stopReason: cancelled, no summarize.
  • handoff_fires_on_each_over_threshold_turn, max_handoffs_zero_disables_end_of_turn_handoff, failed_end_of_turn_summarize_leaves_history_intact replace the per-turn-cap tests.

Mutants caught: is_ok() gate; Reseat::LivePrompt at end of turn; mid-turn gate restored; reply-bytes growth term re-added.

Verification at 7f3b63ea: cargo test -p buzz-agent full package green (456 unit / 54 regressions); fmt + clippy -D warnings clean; pre-push rust-tests + desktop-tauri-checks green.

Independent review: Wren — Minimalness 9/10, Elegance 9/10, Correctness 9/10 at 7f3b63ea, no findings (full suite re-run in a clean worktree).
Live e2e (Max): built from a clean worktree at 7f3b63ea, real OpenAI gpt-5.5 through a recording proxy. 4,000-token window: request 1 reported 4,701 input tokens, 14 KB tool result, turn completed with no mid-turn handoff; summarize + handoff #1 only after the final response; next turn's provider history was exactly [Context Handoff] + new prompt. Cancel mid-provider-call → cancelled, no handoff. 8,000-token window: 1,154 output tokens appended on a turn ending at measured input 6,413 → no handoff. Receipt: .scratch/eot-live-7f3b63ea/ on the dev Mac.

Meli added 2 commits August 22, 2026 18:10
The proactive handoff gate ran at the start of every tool-loop round.
That is the worst moment to compact: the turn's working set (file
reads, tool results) is exactly what the model is using, and summarizing
it away mid-task forces a re-read storm that refills the window. The
round-start gate also projected history growth at a conservative
bytes-per-token rate, so one large tool result could trip it on its own.

Run the gate once, after `run()` returns, so the NEXT turn starts on a
fresh context and the current turn keeps what it was working with.

- handoff.rs: `maybe_handoff(&mut attempts)` → `end_of_turn_handoff()`.
  Gates on `max_handoffs == 0` (disabled) and `should_handoff()`, which
  reads the final successful request's measured usage. `handoff()` takes
  a `Reseat` flag: end of turn does not re-append the already-answered
  prompt; the reactive context-400 path keeps `Reseat::LivePrompt`
  because it continues the same turn.
- agent.rs: drop the per-turn `handoff_attempts` counter and the round-
  start gate; `truncate_history` is now unconditional after
  `drain_steers()`. Mid-turn overflow is handled only by the reactive
  context-400 ladder.
- lib.rs: call `end_of_turn_handoff()` after `ctx.run()` when the result
  is `Ok` and the stop reason is not `Cancelled` — `run()` reports
  cancellation as `Ok(StopReason::Cancelled)`, so an `is_ok()` gate
  would spend a summarize round trip the user just asked us to stop.
- config.rs / README.md: `BUZZ_AGENT_MAX_HANDOFFS` is now a switch —
  0 disables proactive compaction, any positive value enables it; the
  gate runs at most once per turn by construction so the magnitude no
  longer bounds anything. Reactive recovery bypasses it as before.
- The 90% threshold is unchanged; timing and threshold are not tuned
  in the same change.

Tests (tests/regressions.rs) rewritten for the new boundary:
`token_usage_over_budget_triggers_handoff` asserts the summarize lands
inside the over-threshold turn and that the next request is
[handoff block, new prompt] with no re-seated answered prompt;
`history_growth_mid_turn_does_not_trigger_handoff` proves a 6 KB tool
result mid-turn fires nothing and that the gate reads the final
request's usage, not the turn's cumulative input;
`cancelled_turn_skips_end_of_turn_handoff` cancels during an in-flight
provider call; `handoff_fires_on_each_over_threshold_turn`,
`max_handoffs_zero_disables_end_of_turn_handoff`, and
`failed_end_of_turn_summarize_leaves_history_intact` replace the
per-turn-cap tests. Existing handoff fixtures shift one request because
the summarize now lands at the end of the seeding turn.

Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Review finding (Wren): `should_handoff()` still routed its measured-usage
arm through `projected_handoff_input_tokens()`, which added a 1-byte/token
estimate of history appended since the usage was recorded. At end of turn
the only such history is the final assistant reply, so an under-threshold
turn could compact merely because its answer was long (8,500 measured +
a 500-byte reply crossed a 9,000 gate). That contradicted the documented
contract ("final request used 90%") and kept the estimator this change
set out to retire.

- handoff.rs: the `Some(measured_tokens)` arm compares the provider's
  count directly against `token_threshold`. `projected_handoff_input_tokens`
  is deleted; the handoff log line's "before" count uses the measurement
  when present, else the byte-derived bound. The `None` byte fallback is
  unchanged. Doc comments no longer claim "nothing is appended after" the
  final request.
- agent.rs / lib.rs: remove `last_request_history_bytes` (RunCtx field,
  Session field, acquire/run_prompt plumbing, the two clears) — it existed
  only to feed the estimator.
- tests/regressions.rs: add `large_final_reply_does_not_trigger_end_of_turn_handoff`
  (usage 8500 < 9000, 2 KB reply, exactly one request). Fails on the
  parent commit; passes here. Reword
  `reactive_reset_clears_usage_baseline_so_the_gate_is_not_blind` — the
  scenario still holds (a stale sub-threshold reading blinds the gate),
  the "paired bytes / grown" mechanics it described do not.

Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
@tlongwell-block
tlongwell-block requested a review from a team as a code owner August 22, 2026 22:47

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carl, an automated reviewer, commenting via Wes’s GitHub account.

I reviewed exact head 7f3b63ea7fc79ce3881c6a78931325436ea28f29 against base 4baccd5394d6166bb68ff03b24e376e322281a59. Requesting changes for two user-visible correctness defects introduced by moving the handoff await to the turn boundary.

  1. The final-request gate can use an earlier round’s usage. agent.rs preserves last_request_input_tokens when a response omits usage, then should_handoff() treats that retained value as the final request’s measurement. In a two-round tool turn where round 1 reports 950/1000 tokens and the terminal response has no usage block, the agent performs an unwanted summary (three provider requests instead of two). This contradicts the implementation and README contract that the turn’s final request controls the gate, and can either compact an under-threshold final context or miss compaction after an over-threshold final request. Track whether the current/final request reported usage; on absence use the documented byte fallback rather than a prior round, and add this multi-round regression.

  2. Steers accepted during the synchronous end-of-turn summary are silently lost. run_prompt() awaits end_of_turn_handoff() before clearing busy, active_run_id, and steer_tx. Once ctx.run() has returned there is no next round to call drain_steers(), but a steer arriving while the summary is in flight is still accepted with {runId,messageId} and queued into that orphaned receiver. I reproduced this with a hanging summarize request: the late steer returned success and could never reach provider history. Stop advertising/accepting steers before awaiting the post-turn summary, or explicitly persist and route them to a subsequent turn; add a regression around this interval.

Non-blocking test gap: failed_end_of_turn_summarize_leaves_history_intact says the next over-threshold turn retries, but its second terminal response reports usage 10, so it only proves history preservation, not retry. Use a second over-threshold response and assert a second summarize attempt.

Focused validation at exact head:

  • final-response-missing-usage repro: failed, 3 captured requests vs expected 2;
  • steer-during-handoff repro: failed because the completed turn accepted the steer instead of rejecting/routing it;
  • existing focused steer test remained green under the minimal state-lifetime correction I used to validate the seam.

GitHub CI is green and the PR is mergeable, but those checks do not cover these boundary races/contracts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants