Skip to content

feat(trust): publish the per-version fidelity matrix from CI's own reports#74

Open
danfry1 wants to merge 3 commits into
mainfrom
feat/fidelity-dashboard
Open

feat(trust): publish the per-version fidelity matrix from CI's own reports#74
danfry1 wants to merge 3 commits into
mainfrom
feat/fidelity-dashboard

Conversation

@danfry1

@danfry1 danfry1 commented Jul 5, 2026

Copy link
Copy Markdown
Owner

Problem

The RN × Vitest matrix runs the mock-vs-real-RN cross-check for every cell on every commit — and throws the per-cell reports away. The published fidelity page is a single-version snapshot rendered locally, so the project's strongest trust evidence (per-version parity, produced by CI itself) was invisible.

Change

  • Matrix cells upload their cross-check report.json as artifacts (if: always() — a diverging cell's report is the forensic record; 30-day retention).
  • New renderer scripts/fidelity-matrix.mjs aggregates per-cell reports into website/guide/fidelity-matrix.md: an RN × Vitest summary table (resolved versions from the reports themselves) plus a per-probe divergence table. Every report field renders through a shared escape covering the VitePress hazards (pipes, tags, Vue interpolation) and backticks — reports are CI artifacts and are treated as untrusted data end to end.
  • pages.yml re-deploys when the matrix completes on main (workflow_run) and, before building, downloads the latest successful non-pull_request matrix run's artifacts and regenerates the page in the deploy workspace. Nothing generated is committed; the committed page is a placeholder that keeps local builds and dead-link checks working, now drift-gated by fidelity:check so real data can never land in the repo unnoticed.
  • crosscheck.mjs records the resolved Vitest version (the matrix's second axis); the fidelity report page links to the matrix; sidebar entry added.

Security

The pre-PR adversarial review caught a real artifact-poisoning vector in the first draft: selecting runs by head-branch name would have let a fork PR (fork default branch is main) with a modified workflow publish attacker-controlled content to the docs site. The selection now excludes pull_request runs entirely — only push/schedule/dispatch runs, which execute trusted main-branch code, are eligible — and the deploy job skips fork-triggered workflow_run events outright. All rendered report fields are escaped as untrusted input as defense in depth.

Behavior notes

  • A failed matrix run (any divergence) is never published — the site keeps showing the last all-green run while CI is red; the divergence table renders only for manually-supplied report directories (documented in-code).
  • With no eligible run or expired artifacts, the placeholder ships — the site build never depends on matrix availability.

Tests

7 tests execute the real script: placeholder render, numeric-sorted aggregation, per-cell rows, escaping of reasons AND hostile probe names (backtick/pipe/HTML injection), artifact-name fallback, corrupt-JSON failure (placeholder ships), and the --check drift gate both ways.

Full gate: mock suite 1299, crosscheck 75/75 (report now carries vitestVersion), fidelity:check (both pages), website build, YAML lint, lint/format/typecheck clean.

danfry1 added 2 commits July 5, 2026 14:45
…ports

The RN × Vitest matrix has always run the mock-vs-real-RN cross-check per
cell and discarded the per-cell reports. Those reports are now uploaded as
artifacts (on failure too — a diverging cell's report is the forensic
record) and aggregated into a published dashboard:

- scripts/fidelity-matrix.mjs renders website/guide/fidelity-matrix.md from
  a directory of per-cell report.json files: a React Native × Vitest summary
  table plus a per-probe divergence table, with corpus text escaped against
  the VitePress hazards (pipes, bare tags, Vue interpolation). With no
  reports it renders the committed placeholder, so local site builds and
  dead-link checks never depend on matrix availability.
- pages.yml gains a workflow_run trigger (matrix completed on main) and a
  best-effort pre-build step that downloads the latest successful matrix
  run's report artifacts and regenerates the page in the deploy workspace —
  the site refreshes with every matrix run without committing generated
  data. Failed matrix runs are deliberately not published: the site shows
  the last green state while CI is red.
- crosscheck.mjs records the resolved Vitest version in the ephemeral report
  (the matrix's second axis), and the fidelity report page links to the
  matrix.

The dashboard turns 'CI runs this range on every commit' from a claim into
a page generated by the runs themselves.
…scape all report fields

Pre-PR review findings:

- BLOCKER: the artifact-selection query filtered by head branch name, and
  the matrix workflow runs on pull_request — a fork PR opened from a branch
  named 'main' (the fork default) would qualify as 'the latest successful
  main run', and pull_request runs execute the PR's own workflow file, so
  its artifacts are attacker-controlled and would be rendered into the
  published site. The query now selects only non-pull_request runs
  (push/schedule/workflow_dispatch execute trusted main-branch code), and
  the deploy job skips workflow_run triggers fired by pull_request matrix
  runs outright.
- Probe names were the one report field rendered unescaped (inside a code
  span a backtick breaks out of); every report field now renders through the
  shared escape, which additionally neutralizes backticks.
- Documented that the divergence table is unreachable from CI-selected data
  (crosscheck exits non-zero on divergence, so failed runs are never
  selected) — it renders for manually-supplied report directories.
- fidelity:check now also gates the committed fidelity-matrix.md as exactly
  the placeholder render, so real data can never be committed unnoticed.

Tests: hostile probe name (backtick/pipe/HTML injection), corrupt artifact
JSON fails the render step (placeholder ships), --check pass/fail.
Comment thread packages/vitest-native/scripts/fidelity-matrix.mjs Fixed
CodeQL (js/incomplete-sanitization): escaping pipes/tags without first
escaping backslashes lets an input backslash re-arm the escaped character
('a\' + '|' rendered a literal backslash followed by a LIVE pipe, breaking
the table row). Both generated-page escapers (fidelity-matrix.mjs and
fidelity-report.mjs, which had the same pre-existing gap flagged on main)
now escape backslashes first; the escaping test carries a backslash-payload
case. Regenerated artifacts are byte-identical for the current corpus (no
probe text contains backslashes).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants