Skip to content

fix(cross-model): send grok peers the prompt verbatim on read-only routes - #1339

Merged
tmchow merged 1 commit into
EveryInc:mainfrom
taihartman:fix/grok-verbatim-readonly-peers
Aug 6, 2026
Merged

fix(cross-model): send grok peers the prompt verbatim on read-only routes#1339
tmchow merged 1 commit into
EveryInc:mainfrom
taihartman:fix/grok-verbatim-readonly-peers

Conversation

@taihartman

Copy link
Copy Markdown
Contributor

Problem

grok --prompt-file does not send the file as given. Past roughly 50KB the CLI writes a large-prompt offload file, inlines a bounded preview in its place, and expects the agent to read the remainder back with a file tool. The mechanism is observable in the binary itself (grok 0.2.118 carries the string failed to write large-prompt offload file; sending bounded preview with no file reference). --verbatim is documented as "Send the prompt exactly as given" and turns it off.

Three of the four peer scripts never pass it:

script --verbatim
skills/ce-code-review/scripts/cross-model-adversarial-review.sh missing
skills/ce-doc-review/scripts/cross-model-doc-review.sh missing
skills/ce-pov/scripts/cross-model-pov.sh missing
skills/ce-work/scripts/cross-model-work.sh present

On ce-doc-review, which passes --deny Read, the offloaded prompt is unrecoverable. The observed symptom is a peer that reviews only the tail of the document and returns confident, schema-valid findings about it — or, in one run, findings quoting text that does not exist in the document at all. Neither failure is visible downstream: the artifact is well-formed and carries independence_verified: true.

On ce-code-review and ce-pov, which do allow Read, it is a degradation rather than a guaranteed loss: the offload file lands outside the --cwd they pin, and any recovery attempt spends turns against a tight --max-turns on a --json-schema route that returns nothing at all if it runs out.

Why it was missed

ce-work already has the flag, and it also grants --tools Read,Write,Edit, so it is safe twice over and could recover an offloaded prompt without the flag. The flag landed first on the one route whose permissions made it least necessary, and was never carried to the stricter reviewers, where it is the only thing standing between a peer and a review of content it never received.

The fix is the flag, not weaker isolation. --deny Read stays exactly as it is.

Verification

Isolated A/B, grok-4.5, identical flags, question placed at the head of the payload:

prompt size --verbatim result
17,637 B no correct (below the offload threshold)
52,302 B no peer narrates "the direct read was blocked", answers NO_DOCUMENT
52,302 B yes correct

End-to-end through cross-model-doc-review.sh itself, same lens and document, measuring where in the document each returned evidence quote actually appears:

arm quotes verified present position in document
stock 6/6 96%, 97%, 98%, 99%, 99%, 96%
patched 11/11 11%–51%

The control arm's quotes are all genuine. They are all drawn from the final 4% of the file, because that is all the peer received.

Changes

--verbatim added to the grok-cli adapter in the three scripts that lack it, with a rationale comment at each site. Each script's existing route-safety test now pins the flag; all three assertions fail against the pre-fix scripts.

bun run test 2908 pass / 0 fail across 110 files. release:validate and plugin:validate --strict both pass.

Follow-up, not included here

Neither failure shape is detectable at synthesis today. An evidence-quote existence check would catch the fabricating run and miss the truncating one, since its quotes are real. A positional check — evidence clustering in the tail of the document means the peer never saw the body — would catch both. That is a design change to synthesis rather than a one-flag fix, so it is left for a separate issue.

…utes

`grok --prompt-file` does not send the file as given by default. Past roughly
50KB the CLI writes a large-prompt offload file, inlines a bounded preview in
its place, and expects the agent to read the remainder back with a file tool.
`--verbatim` ("Send the prompt exactly as given", grok 0.2.118) is what turns
that off.

The three peer scripts missing the flag are the read-only reviewers, and they
are exactly the ones that cannot survive the offload:

- ce-doc-review passes `--deny Read`, so the offloaded prompt is
  unrecoverable. Observed symptom: the peer narrates that the read was
  blocked and returns findings quoting text that does not exist in the
  reviewed document.
- ce-code-review and ce-pov do allow Read, but the offload file lives outside
  the `--cwd` they pin, and any recovery attempt burns turns against a tight
  `--max-turns` budget on a `--json-schema` route that returns nothing at all
  if it runs out.

skills/ce-work/scripts/cross-model-work.sh is the in-tree precedent — it
already passes `--verbatim`, and it is safe twice over because it also grants
`--tools Read,Write,Edit` and could therefore recover an offloaded prompt
without the flag. That inversion is why this was missed: the flag landed
first on the one route whose permissions made it least necessary, and was
never carried to the stricter reviewers, where it is the only thing standing
between a peer and a fabricated review. The tighter the sandbox, the more
essential `--verbatim` is.

The fix is the flag, not weaker isolation — `--deny Read` stays as it is.

Each route-safety test now pins `--verbatim` on its grok-cli adapter; all
three fail against the pre-fix scripts.
@tmchow
tmchow merged commit 53f7b9a into EveryInc:main Aug 6, 2026
3 checks passed
@github-actions github-actions Bot mentioned this pull request Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants