fix(cross-model): send grok peers the prompt verbatim on read-only routes - #1339
Merged
Merged
Conversation
…utes
`grok --prompt-file` does not send the file as given by default. Past roughly
50KB the CLI writes a large-prompt offload file, inlines a bounded preview in
its place, and expects the agent to read the remainder back with a file tool.
`--verbatim` ("Send the prompt exactly as given", grok 0.2.118) is what turns
that off.
The three peer scripts missing the flag are the read-only reviewers, and they
are exactly the ones that cannot survive the offload:
- ce-doc-review passes `--deny Read`, so the offloaded prompt is
unrecoverable. Observed symptom: the peer narrates that the read was
blocked and returns findings quoting text that does not exist in the
reviewed document.
- ce-code-review and ce-pov do allow Read, but the offload file lives outside
the `--cwd` they pin, and any recovery attempt burns turns against a tight
`--max-turns` budget on a `--json-schema` route that returns nothing at all
if it runs out.
skills/ce-work/scripts/cross-model-work.sh is the in-tree precedent — it
already passes `--verbatim`, and it is safe twice over because it also grants
`--tools Read,Write,Edit` and could therefore recover an offloaded prompt
without the flag. That inversion is why this was missed: the flag landed
first on the one route whose permissions made it least necessary, and was
never carried to the stricter reviewers, where it is the only thing standing
between a peer and a fabricated review. The tighter the sandbox, the more
essential `--verbatim` is.
The fix is the flag, not weaker isolation — `--deny Read` stays as it is.
Each route-safety test now pins `--verbatim` on its grok-cli adapter; all
three fail against the pre-fix scripts.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
grok --prompt-filedoes not send the file as given. Past roughly 50KB the CLI writes a large-prompt offload file, inlines a bounded preview in its place, and expects the agent to read the remainder back with a file tool. The mechanism is observable in the binary itself (grok 0.2.118 carries the stringfailed to write large-prompt offload file; sending bounded preview with no file reference).--verbatimis documented as "Send the prompt exactly as given" and turns it off.Three of the four peer scripts never pass it:
--verbatimskills/ce-code-review/scripts/cross-model-adversarial-review.shskills/ce-doc-review/scripts/cross-model-doc-review.shskills/ce-pov/scripts/cross-model-pov.shskills/ce-work/scripts/cross-model-work.shOn
ce-doc-review, which passes--deny Read, the offloaded prompt is unrecoverable. The observed symptom is a peer that reviews only the tail of the document and returns confident, schema-valid findings about it — or, in one run, findings quoting text that does not exist in the document at all. Neither failure is visible downstream: the artifact is well-formed and carriesindependence_verified: true.On
ce-code-reviewandce-pov, which do allow Read, it is a degradation rather than a guaranteed loss: the offload file lands outside the--cwdthey pin, and any recovery attempt spends turns against a tight--max-turnson a--json-schemaroute that returns nothing at all if it runs out.Why it was missed
ce-workalready has the flag, and it also grants--tools Read,Write,Edit, so it is safe twice over and could recover an offloaded prompt without the flag. The flag landed first on the one route whose permissions made it least necessary, and was never carried to the stricter reviewers, where it is the only thing standing between a peer and a review of content it never received.The fix is the flag, not weaker isolation.
--deny Readstays exactly as it is.Verification
Isolated A/B, grok-4.5, identical flags, question placed at the head of the payload:
--verbatimNO_DOCUMENTEnd-to-end through
cross-model-doc-review.shitself, same lens and document, measuring where in the document each returned evidence quote actually appears:The control arm's quotes are all genuine. They are all drawn from the final 4% of the file, because that is all the peer received.
Changes
--verbatimadded to thegrok-cliadapter in the three scripts that lack it, with a rationale comment at each site. Each script's existing route-safety test now pins the flag; all three assertions fail against the pre-fix scripts.bun run test2908 pass / 0 fail across 110 files.release:validateandplugin:validate --strictboth pass.Follow-up, not included here
Neither failure shape is detectable at synthesis today. An evidence-quote existence check would catch the fabricating run and miss the truncating one, since its quotes are real. A positional check — evidence clustering in the tail of the document means the peer never saw the body — would catch both. That is a design change to synthesis rather than a one-flag fix, so it is left for a separate issue.