feat(ci): add isolated image qualification harness - #1775
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
|
@clawsweeper review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
Codex review: needs real behavior proof before merge. Reviewed September 4, 2026, 10:44 AM ET / 14:44 UTC. ClawSweeper reviewWhat this changesAdds isolated GitHub Actions workflows, a private relay/controller, scripts, tests, and documentation for qualifying an exact image-publishing candidate against a credential-isolated AWS sandbox. Merge readiness⛔ Blocked before merge - 6 items remain Keep open. This MEMBER-authored PR adds a new, privileged CI and AWS qualification capability; its source-level isolation design is coherent, but the required deployed authority-chain proof and maintainer decision to operate the protected environment are still absent. Priority: P1 Review scores
Verification
How this fits togetherThe image qualification subsystem runs a candidate Crabbox build through a protected workflow and a narrowly scoped relay. The relay forwards only approved publisher calls to a private candidate Worker, which reaches the existing AWS qualification authority and produces cleanup evidence. flowchart LR
A[Candidate pull request] --> B[Protected qualification workflow]
B --> C[Sealed candidate bundle]
C --> D[Private candidate Worker]
E[Credentialless executor] --> F[Scoped public relay]
F --> D
D --> G[AWS qualification authority]
G --> H[Rollback and cleanup evidence]
Decision needed
Why: This choice creates a persistent privileged CI and paid-provider operating surface; source review cannot decide whether that operational commitment is wanted. Before merge
Findings
Agent review detailsSecurityNeeds attention: No concrete source-level bypass was found, but the new relay-to-candidate authority injection has no deployed final-effect proof. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Provision the reviewed protected environment and sandbox only after maintainer sponsorship, then retain this harness once a redacted exact-candidate run proves rollback recovery, final-effect rejection, and zero-residue teardown. Do we have a high-confidence way to reproduce the issue? Not applicable as a bug reproduction. The required feature scenario has not been run in its real protected GitHub, Cloudflare, and AWS environment. Is this the best way to solve the issue? Unclear. The isolation approach is source-consistent, but it is not yet the proven best solution until maintainers accept the operating model and a deployed run validates the final authority boundary. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against b061bb5dbdb7. LabelsLabel changes:
Label justifications:
EvidenceSecurity concerns:
What I checked:
Likely related people:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (18 earlier review cycles; latest 8 shown)
|
fc3f7b4 to
240f169
Compare
|
@clawsweeper re-review Please review exact head |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review Please review exact head The prior CodeQL findings are repaired by moving candidate compilation to an unprivileged |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review Please review exact head This head repairs the exact admission finding: a credentialless trusted job now verifies the immutable artifact and rejects publishers without the ordered transactional rollback contract before protected deployment. The protected deploy repeats the check before its first side effect, and the contract admits the actual PR 1756 publisher. |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review Please review exact head This head repairs the remaining exact-head findings:
No workflow, deployment, cloud resource, environment, secret, or paid run was created. |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review Please review exact head This signed follow-up repairs the exact The protected-path deployment proof remains externally blocked and was not |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review Exact head: This head makes the candidate Worker private, adds a bounded trusted relay with |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
afadad8 to
1ef24dd
Compare
|
@clawsweeper re-review Exact restacked head: The restacked diff remains 10 files and |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: feat(ci): add isolated image qualification harness This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
Related: #1756
Stacked on: #1778
What Problem This Solves
Image publication changes need exact-candidate proof of Fleet auth, AWS lifecycle,
transactional catalog rollback, and zero cloud residue before merge. Running
candidate code in a credential-bearing job or the default-branch cache scope
would make that proof unsafe.
Why This Change Was Made
This replaces the earlier privileged harness with a credential-isolated
workflow stacked on the AWS qualification authority. The protected
default-branch
workflow_dispatchowns immutable credentialless build commandsand explicitly checks out both the protected revision and authorized candidate
SHA. It rejects candidate changes to Go modules or non-source Worker inputs,
installs only the protected lockfile, copies a bounded regular-file
worker/srctree into the protected build root, and runs only the protected Goand Wrangler toolchains. A manifest-covered receipt binds the candidate source
and protected build-input digests; later jobs accept only that artifact ID and
digest from the exact first-attempt workflow run.
Before environment approval, a credentialless trusted admission job verifies
the exact artifact and rejects publishers without injectable CLI delegation,
pre-promotion candidate teardown, transactional receipts, compare-and-swap
rollback, and failed-revision retirement. Protected jobs execute only trusted
control tooling and treat candidate bytes as inert deploy data.
Protected deployment first creates a registry-discoverable inert candidate
bootstrap, disables workers.dev and preview URLs, and only then uploads exact
candidate bytes. A separate trusted relay exposes the sole workers.dev endpoint
and has one service binding to the private candidate. Its fixed allowlist
reconstructs only the publisher proof's lease, image, and run requests and
rejects arbitrary targets, paths, methods, queries, header overrides, content
types, oversized bodies, and absent or expired run timestamps before candidate
dispatch; expiry is rechecked immediately before the service-binding call.
After deployment and enrollment, a final protected
armjob revalidates theopen PR, candidate artifact, exact candidate and relay Worker versions, binding
settings, isolation, registry claim, authority identity, policy hash, and
enrollment timestamps. A canonical execution-manifest digest binds both final
Worker versions to the candidate/deployment/authority identity before the
credentialless execute job can make its first mutation or paid call.
The executor receives only the relay URL and a distinct ephemeral executor
token. Candidate admin/shared tokens stay inside the trusted relay; AWS signing
remains in the fixed-policy authority Worker. Cloudflare and controller
credentials never enter candidate jobs or bindings. An independent protected
reaper derives and deletes the exact relay and candidate from durable run
identity without workflow artifacts.
User Impact
The reviewer-gated proof covers:
source-derived current-run artifact with protected build-input receipt,
private candidate deployment, and exact final candidate/relay Worker versions;
bodies, header smuggling, wrong executor tokens, and expired runs before
candidate dispatch, including a final expiry recheck at the dispatch boundary;
promote-casdenial bounded to the exact millisecondrequest window, with unchanged candidate API readback and no operation or signer
timestamp inside that inclusive interval;
three total;
R0, candidateR1,and a fresh base rollback
R2, withR2distinct fromR0andR1;R1retirement, accepting a200readback only for thesame AMI as a provider-only record with no catalog identity or role fields;
R0CAS returning 409 with currentR2, followed by unchanged baseand failed-image readbacks; logs are supplemental only;
credentialless executor and protected zero-residue finalization.
PR/head/base drift is recorded as a qualification failure, never a warning.
Cleanup remains unconditional. Protected teardown first persists the authority
and registry finalization fence, then disables, verifies, and deletes the public
relay before continuing cleanup/finalization. The protected final verifier
matches run ID, candidate SHA, deployment hash, authority SHA/version, policy
hash, enrollment and expiry timestamps, finalized state, exact candidate/relay
Worker versions, and both binding digests before retiring the registry.
Limits remain fixed to one attempt, one active instance,
t3.small/t3a.smallon-demand, one image/checkpoint set, no FSR or instanceprofile, $10, and an absolute 120-minute authority expiry.
Evidence
node --test --test-name-pattern='control tool fixes|executor proves|attestation gate|catalog proof|relay rejects' scripts/image-qualification-workflow.test.js- 5 passed1ef24dd717b46991e9631082decd62a45ca87c02:node --test scripts/image-qualification-workflow.test.js- 17 passedb6b4be90113f3d8930083e2f6ca836a18c81e296source and contract inspection for transactional promotion receipts, fresh revisions, and 409 CAS conflictsactions/upload-artifact/download-artifactsource and docsinspection, including bare upload digest normalization and current-run ID downloads
configuration; qualification uses the protected config and binary only
artifact replacement while retaining exact repo/PR/SHA checks
node --test scripts/workflow-action-pins.test.js- 1 passedactionlint -shellcheck= -pyflakes= .github/workflows/image-qualification.yml .github/workflows/image-qualification-reaper.ymlbuild-candidateshell:shellcheck -xandbash -nshellcheck scripts/image-qualification-execute.sh scripts/image-qualification-crabbox-adapter.shnode --check scripts/image-qualification-control.mjsnode --check scripts/image-qualification-workflow.test.jsnode --check scripts/image-qualification-controller-worker.mjsnode --check scripts/image-qualification-relay-worker.mjsbash -n scripts/image-qualification-execute.sh scripts/image-qualification-crabbox-adapter.shgit diff --check1ef24dd717b46991e9631082decd62a45ca87c02b061bb5dbdb776c3c11b538204034184eb6dd8a7after squash-merging PR 1778No workflow was dispatched and no GitHub environment, secret, Worker, Durable
Object, AWS resource, deployment, or paid run was created.
External setup blockers for live proof:
image-qualificationGitHub environment;account with the documented region/tag IAM constraints and explicit FSR deny;
CLOUDFLARE_API_TOKEN,CRABBOX_IMAGE_QUALIFICATION_CONTROLLER_TOKEN,CLOUDFLARE_ACCOUNT_ID, theauthority SHA/version and canonical policy hash, AWS region, subnet,
preprovisioned security group, base AMI, and root-volume variables;
PR 1756 candidate after the protected environment and authority are provisioned.
Production/operations delta: +2847/-0 lines. Tests: +1094/-0 lines. Docs: +147 net (+155/-8) lines.
Positive production growth implements the credential and source-derived build
boundaries, immutable deployment contract, protected recovery owner, private candidate ingress,
bounded trusted relay, exact catalog-state proof, pre-execution version fence,
and zero-residue evidence surface; normal CLI and coordinator behavior are
unchanged.