Repository navigation
Let the model replace a junk tool result with a one-line summary (model-judged context pruning) #5362
Description
Describe the feature or problem you'd like to solve
TL;DR: When a tool call returns junk the model didn't intend, let the model swap that output for a one-line summary before the turn ends.
Pay for it once; keep the context clean for the rest of the session.
Important
Why prevention can't fix this
You open a file or run a command because you don't know what's in it, so its output size can't be predicted in advance.
A human glances at junk and looks away. A model re-reads it on every turn.
This gives the model the ability to look away. 🙈
Proposed solution
Sequence
TODAY
turn N grep -r ... ──► ~40 KB of target/ build fingerprints
model ──► "oops, irrelevant" (but can't drop it)
turn N+1..50 every request ──► re-sends the same ~40 KB ❌ cost + diluted attention
PROPOSED
turn N grep -r ... ──► ~40 KB
model ──► prune_tool_result(call_id, "grep hit target/ fingerprints, irrelevant")
context now ──► [pruned] grep hit target/ fingerprints, irrelevant (1 line)
turn N+1..50 every request ──► 1 line ✅
Why it's cheap: only the tail changes
[ instructions | turns 1 .. N-1 | junk output | rest of turn N ]
└───── cached prefix: kept ──────┘ └── re-cached once (small) ──┘
Prompt caching is prefix-based, so pruning a result from the current turn leaves everything before it cached.
Design
| Rule | Why |
|---|---|
| Model-judged | "This is junk" is a semantic call; no size cap or rule can make it. |
| Tail-only (current turn) | The cached prefix survives; nothing old is rewritten. |
| Visible | Original stays in the user-facing transcript with a one-line notice. Nothing disappears silently. |
Bonus: a bigger effective window
Removing junk grows the useful space by more than the junk's share, because the denominator shrinks:
| Junk share of context | Useful space | Effective window gain |
|---|---|---|
| 10% | 90% → 100% | +11% |
| 20% | 80% → 100% | +25% |
| 30% | 70% → 100% | +43% |
| 50% | 50% → 100% | +100% |
Same hardware, same model: compaction comes later, and what remains is signal. (Junk shares are illustrative.)
Example prompts or workflows
grep -rthat unexpectedly walks a build/output directory (target/,node_modules/,dist/)catof a file that turns out to be generated or minified- a full
pytest/npm install/piplog when only the last error line mattered
In each case the model knows within one glance that the output is noise, but it is carried in every later request.
Additional context
| Mechanism | Gap |
|---|---|
| Automatic compaction (with checkpoints) | Periodic and lossy; junk rides along until it fires, and the summary can drop what was needed. |
| Output size limits | Cut by size: wanted long output is cut, short-enough junk passes. A grep can be essential one call and junk the next. |
"This output is junk" is a semantic judgment only the model can make, at the moment the output arrives.
Everyone wins 🌱
| Who | What they get |
|---|---|
| Users | Fewer tokens burned per task |
| The model | A clean context, better answers |
| Providers | Less KV cache per session, more sessions per GPU |
| The planet | Less power, fewer GPUs to build |
Junk is pure overhead: no answer improves because a build-fingerprint dump was re-read many times. Drop it in every harness, and the whole industry does the same work with less. The cheapest watt is the one you never spend.
Filed across 17 agent harnesses; full list: https://gist.github.com/nicezic/69f13fde6fa080a571825b56fd76392d
Activity
nicezic commentedon Nov 5, 2026
Rough impact estimate (Fermi: every number is an assumption)
TL;DR: a single unnoticed junk result is small. Multiplied by every session and every turn it rides along, it's large.
Per session
Fleet-wide
Who gains
Caveats
(If this ships, I'll happily accept a lifetime GitHub Copilot Pro+ plan as payment. Just kidding, sorry! 😄)
nicezic commentedon Nov 5, 2026
Quality, not just cost
TL;DR: clean context makes better agents. Model-judged pruning removes the noise that rule-based clearing keeps.
Evidence that context hygiene moves results
Anthropic's context management post reports, on an internal agentic-search eval:
Where junk comes from in autonomous runs (SWE-bench / Terminal-Bench style)
The longer the task, the more the noise competes with the signal, and long-horizon tasks are where agent benchmarks are won or lost.
Why model-judged beats rules here
pytestoutput aboveprune_tool_result)Suggested A/B
prune_tool_resultMetrics: resolve rate, tokens per task, and over-prune rate (how often the model pruned something it later needed; this is the main risk to measure).