feat(trust): publish the per-version fidelity matrix from CI's own reports#74
Open
danfry1 wants to merge 3 commits into
Open
feat(trust): publish the per-version fidelity matrix from CI's own reports#74danfry1 wants to merge 3 commits into
danfry1 wants to merge 3 commits into
Conversation
…ports The RN × Vitest matrix has always run the mock-vs-real-RN cross-check per cell and discarded the per-cell reports. Those reports are now uploaded as artifacts (on failure too — a diverging cell's report is the forensic record) and aggregated into a published dashboard: - scripts/fidelity-matrix.mjs renders website/guide/fidelity-matrix.md from a directory of per-cell report.json files: a React Native × Vitest summary table plus a per-probe divergence table, with corpus text escaped against the VitePress hazards (pipes, bare tags, Vue interpolation). With no reports it renders the committed placeholder, so local site builds and dead-link checks never depend on matrix availability. - pages.yml gains a workflow_run trigger (matrix completed on main) and a best-effort pre-build step that downloads the latest successful matrix run's report artifacts and regenerates the page in the deploy workspace — the site refreshes with every matrix run without committing generated data. Failed matrix runs are deliberately not published: the site shows the last green state while CI is red. - crosscheck.mjs records the resolved Vitest version in the ephemeral report (the matrix's second axis), and the fidelity report page links to the matrix. The dashboard turns 'CI runs this range on every commit' from a claim into a page generated by the runs themselves.
…scape all report fields Pre-PR review findings: - BLOCKER: the artifact-selection query filtered by head branch name, and the matrix workflow runs on pull_request — a fork PR opened from a branch named 'main' (the fork default) would qualify as 'the latest successful main run', and pull_request runs execute the PR's own workflow file, so its artifacts are attacker-controlled and would be rendered into the published site. The query now selects only non-pull_request runs (push/schedule/workflow_dispatch execute trusted main-branch code), and the deploy job skips workflow_run triggers fired by pull_request matrix runs outright. - Probe names were the one report field rendered unescaped (inside a code span a backtick breaks out of); every report field now renders through the shared escape, which additionally neutralizes backticks. - Documented that the divergence table is unreachable from CI-selected data (crosscheck exits non-zero on divergence, so failed runs are never selected) — it renders for manually-supplied report directories. - fidelity:check now also gates the committed fidelity-matrix.md as exactly the placeholder render, so real data can never be committed unnoticed. Tests: hostile probe name (backtick/pipe/HTML injection), corrupt artifact JSON fails the render step (placeholder ships), --check pass/fail.
CodeQL (js/incomplete-sanitization): escaping pipes/tags without first
escaping backslashes lets an input backslash re-arm the escaped character
('a\' + '|' rendered a literal backslash followed by a LIVE pipe, breaking
the table row). Both generated-page escapers (fidelity-matrix.mjs and
fidelity-report.mjs, which had the same pre-existing gap flagged on main)
now escape backslashes first; the escaping test carries a backslash-payload
case. Regenerated artifacts are byte-identical for the current corpus (no
probe text contains backslashes).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The RN × Vitest matrix runs the mock-vs-real-RN cross-check for every cell on every commit — and throws the per-cell reports away. The published fidelity page is a single-version snapshot rendered locally, so the project's strongest trust evidence (per-version parity, produced by CI itself) was invisible.
Change
report.jsonas artifacts (if: always()— a diverging cell's report is the forensic record; 30-day retention).scripts/fidelity-matrix.mjsaggregates per-cell reports intowebsite/guide/fidelity-matrix.md: an RN × Vitest summary table (resolved versions from the reports themselves) plus a per-probe divergence table. Every report field renders through a shared escape covering the VitePress hazards (pipes, tags, Vue interpolation) and backticks — reports are CI artifacts and are treated as untrusted data end to end.pages.ymlre-deploys when the matrix completes on main (workflow_run) and, before building, downloads the latest successful non-pull_request matrix run's artifacts and regenerates the page in the deploy workspace. Nothing generated is committed; the committed page is a placeholder that keeps local builds and dead-link checks working, now drift-gated byfidelity:checkso real data can never land in the repo unnoticed.crosscheck.mjsrecords the resolved Vitest version (the matrix's second axis); the fidelity report page links to the matrix; sidebar entry added.Security
The pre-PR adversarial review caught a real artifact-poisoning vector in the first draft: selecting runs by head-branch name would have let a fork PR (fork default branch is
main) with a modified workflow publish attacker-controlled content to the docs site. The selection now excludespull_requestruns entirely — only push/schedule/dispatch runs, which execute trusted main-branch code, are eligible — and the deploy job skips fork-triggeredworkflow_runevents outright. All rendered report fields are escaped as untrusted input as defense in depth.Behavior notes
Tests
7 tests execute the real script: placeholder render, numeric-sorted aggregation, per-cell rows, escaping of reasons AND hostile probe names (backtick/pipe/HTML injection), artifact-name fallback, corrupt-JSON failure (placeholder ships), and the
--checkdrift gate both ways.Full gate: mock suite 1299, crosscheck 75/75 (report now carries
vitestVersion),fidelity:check(both pages), website build, YAML lint, lint/format/typecheck clean.