Skip to content

docs: fix prompt-audit contract drift and archaeology (v0.32.2) - #196

Open
omkhar wants to merge 2 commits into
mainfrom
audit/prompt-audit-fixes
Open

omkhar wants to merge 2 commits into
mainfrom
audit/prompt-audit-fixes

Conversation

@omkhar

@omkhar omkhar commented Sep 7, 2026

Copy link
Copy Markdown
Owner

Applies the deduplicated High/Medium findings from two 2026-09-06 prompt audits (repo audit + installed-skills audit). One line per finding:

  • H1 (contract bug) — proof-harness-router.md offered demoting a post-admission candidate to referent_unresolved after two failed proof attempts; that contradicts loop-engineering.md (demotion is pre-admission only; an admitted candidate keeps its in-scope row and closes via closure-bundles.json) and the validator, which makes referent_unresolved a candidate-fusion.json decision value and blocks it from closing. Following the router would break the enforced in_scope_count == len(verification-sheet.json) reconciliation identity — a finding would silently vanish, the exact failure the Finding-Completeness gate exists to prevent. The router now states the loop-engineering rule (the side backed by code).
  • High — tool-firewall.md claimed the read-side planner-ingest link "remain[s] a tracked v0.16.x follow-up"; validate_planner_ingest ships and fires since v0.23.0 (repo at 0.32.x). Rewritten to state the enforced behavior. (Both audits flagged this; applied once.)
  • High — tool-firewall.md "built in a subsequent phase — not this reference" fossil on scripts/quarantine_extract.py, which ships today; parenthetical removed.
  • Medium — undefined review-round markers (R1)(R2)(R8)(R10)(R13)(R16)(R25) removed from discovery-intake.md (10 sites, every rule preserved). (Both audits; applied once.)
  • Medium — dangling spec/section identifiers removed: §3.12 in sanitizer-selection.md (pointer now names references/promoted-replay-contract.md), §3.15.8 + R5 in tool-firewall.md, "§3.15.8 replay fields" in controlled-fields.md, "Guardrail A" in surface-tool-inventory.md. Kept B1 and §3.15.6: AST-verified they appear verbatim in validator runtime error strings, so they are live vocabulary mapping gate output to its reference, per the audit's own keep-list criterion.
  • Medium — skill-maintainer build guidance moved (not deleted) from discovery-orchestrator.md, discovery-specialist-router.md ("Deterministic Validator Candidates"), and shadow-skill-bootstrap.md into a new AGENTS.md "Skill Maintenance" section.
  • Medium — migration-relative phrasing rewritten as present-state rules in discovery-specialist-router.md ("Specialist Reviewer Redesign" → "Specialist Reviewers") and artifact-contract.md ("preserve the prior semantics" dropped).
  • Medium — skill.json routing description drops the trailing behavioral imperative ("validate it alongside self-found candidates") that the canonical body and AGENTS.md already mandate; both routing signals kept.

Doc accuracy only — no validator, vocabulary, or workflow-semantics change. make sync regenerated all four adapter packages; make verify green (797 unit + 9 integration). Version bumped to 0.32.2 with CHANGELOG entry and consensus record (reviews/prompt-audit-contract-drift-consensus.md) per the 0.32.1 precedent. proof-harness-router.md size budgets raised 57→59 lines with comments citing the record.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EUo5YkQZdCTkn2bBfY9hou

Two 2026-09-06 prompt audits, deduplicated. The load-bearing fix:
proof-harness-router.md offered demoting a post-admission candidate to
referent_unresolved after an exhausted retry budget, contradicting
loop-engineering.md (demotion is pre-admission only) and the validator
(referent_unresolved is a candidate-fusion decision, blocked from
closing); following it would break the enforced
in_scope_count == len(verification-sheet) identity. The router now
states the loop-engineering rule.

Also: tool-firewall.md drops the false "tracked v0.16.x follow-up"
claim (validate_planner_ingest ships and fires since v0.23.0) and the
"built in a subsequent phase" fossil on quarantine_extract.py; dangling
review-round/spec markers ((R1)..(R25), section 3.12/3.15.8, R5,
Guardrail A) removed from five references while B1/section 3.15.6 stay
(live in validator error strings, AST-verified); skill-maintainer build
guidance moved to a new AGENTS.md Skill Maintenance section;
migration-relative phrasing rewritten as present-state rules; the
skill.json routing description drops the behavioral clause the body
already mandates. Doc accuracy only: no validator, vocabulary, or
workflow-semantics change. Surfaces regenerated; consensus record
added; bump to 0.32.2.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EUo5YkQZdCTkn2bBfY9hou
@omkhar

omkhar commented Sep 7, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 7, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-07T04:25:37.305122Z 365ae54 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dfb6ae2eef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +53 to +55
The five-agent official-docs recheck was run and recorded 2026-09-06 (feature-matrix refresh
date, `docs/authoritative-sources.md`); this change alters no agent envelope, packaging, or
adopted feature, so that one-day-old recheck stands as current.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Record this release's required five-agent refresh

Because this commit substantively changes runtime-loaded references and the skill's routing metadata, reusing the v0.32.1 survey does not satisfy the required same-pass refresh. Lines 53–55 acknowledge that neither docs/authoritative-sources.md nor skill-core/references/agent-feature-matrix.md was updated for v0.32.2, leaving this release without its mandated dated sources, per-agent decisions, and repeatable feature-matrix record. Perform and record the refresh for this change rather than carrying forward the prior release's entry.

AGENTS.md reference: AGENTS.md:L87-L95

Useful? React with 👍 / 👎.

P1: reusing the v0.32.1 survey does not satisfy the AGENTS.md same-pass
refresh requirement for a substantive reference/metadata change. Re-ran
the five-agent official-docs recheck (research date 2026-09-07): all
five rows n/a/unchanged; new dated entry in
docs/authoritative-sources.md, feature-matrix refresh date and per-row
recheck dates bumped, consensus record corrected. One advisory
observation recorded (Gemini CLI announces future replacement by
Antigravity CLI for unpaid tiers; .agents/skills aliasing unchanged).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EUo5YkQZdCTkn2bBfY9hou
@omkhar

omkhar commented Sep 7, 2026

Copy link
Copy Markdown
Owner Author

Round 1 fix (365ae54): accepted the P1 — re-ran the five-agent official-docs recheck for this pass (research date 2026-09-07) instead of carrying the v0.32.1 entry forward. New dated entry with per-agent sources and n/a-unchanged decisions in docs/authoritative-sources.md; agent-feature-matrix refresh date and per-row recheck dates bumped; consensus record corrected. make verify green.

@omkhar

omkhar commented Sep 7, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 365ae54983

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

reproduction rate as `sound_with_limits` limit metadata (never as the pass/fail), and stay
`blocked_pending_calibration` without TSan confirmation. See `references/proof-harness-router.md` Retry/Stop
and §3.12 nondeterminism handling.
and the nondeterminism handling in `references/promoted-replay-contract.md`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Point to actual nondeterminism guidance

For the unpinned-race path described above, this sends the operator to promoted-replay-contract.md for “nondeterminism handling,” but that file only defines generic replay fields and fixed-promotion transcript rules; it never discusses bounded stress loops, reproduction-rate metadata, or blocked_pending_calibration (a repository-wide search finds that handling only in this file). The cleanup therefore replaces a dangling section number with a misleading target, preventing the promised detail from being loaded on demand; remove the pointer or link/add the actual guidance.

AGENTS.md reference: AGENTS.md:L96-L101

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant