fix(ci): report advisor health and close stale E2E checks#6807
Merged
Conversation
cv
added
the
area: security
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe PR adds optional second-opinion artifact validation and publication, lane-aware review comments, E2E coverage reconciliation, sticky-comment cleanup, and idempotent PR-gate recovery behavior with expanded tests. ChangesAdvisor publication and E2E behavior
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant PublishWorkflow
participant ArtifactValidator
participant CommentCLI
participant GitHub
PublishWorkflow->>ArtifactValidator: validate primary and secondary artifacts
ArtifactValidator-->>PublishWorkflow: return secondary validation status
PublishWorkflow->>CommentCLI: publish primary and validated second-opinion results
CommentCLI->>GitHub: update review comment and delete legacy sticky comments
Possibly related issues
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Contributor
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage remains at 96%, unchanged from the branch. TypeScript / code-coverage/cliThe overall coverage in the branch remains at 79%, unchanged from the branch. Show a code coverage summary of the most impacted files.
Updated |
Contributor
PR Review Advisor — InformationalAdvisor assessment: Informational / low confidence E2E guidanceAdvisory only: coverage and selector recommendations are non-authoritative. E2E / PR Gate independently computes and dispatches trusted jobs without consuming this output. Recommended coverage:
This is an automated, non-authoritative review. Findings are inputs to maintainer adjudication. Warnings and optional suggestions do not require a response or follow-up. A human maintainer makes the final merge decision. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tools/pr-review-advisor/workflow-boundary.mts (1)
531-557: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the duplicated forbidden-
withkey list into one canonical constant.The same list
["run-id", "github-token", "repository", "pattern", "merge-multiple"]is inlined for both the primary (Line 531) and secondary (Line 553) download checks. Since this is a security guardrail, an addition to one list that misses the other silently weakens the check for the other download step.As per path instructions: "Derive inventories and limits from a canonical source where possible; flag duplicated lists that can silently drift."
♻️ Suggested shared constant (define once at module scope)
const FORBIDDEN_DOWNLOAD_WITH_KEYS = [ "run-id", "github-token", "repository", "pattern", "merge-multiple", ] as const;- for (const forbidden of ["run-id", "github-token", "repository", "pattern", "merge-multiple"]) { + for (const forbidden of FORBIDDEN_DOWNLOAD_WITH_KEYS) { if (Object.hasOwn(asRecord(download?.with), forbidden)) { errors.push(`Download primary advisor artifact must not set with.${forbidden}`); } }Apply the same substitution at Line 553 for the secondary download loop.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tools/pr-review-advisor/workflow-boundary.mts` around lines 531 - 557, Define a module-scope canonical constant for the forbidden download `with` keys, using the existing five-key list, then update both primary and secondary download validation loops to iterate over that constant instead of separate inline arrays. Keep the existing error messages and validation behavior unchanged.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tools/pr-review-advisor/workflow-boundary.mts`:
- Around line 531-557: Define a module-scope canonical constant for the
forbidden download `with` keys, using the existing five-key list, then update
both primary and secondary download validation loops to iterate over that
constant instead of separate inline arrays. Keep the existing error messages and
validation behavior unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 4e438f9b-4946-4d3e-b999-a32293d81494
📒 Files selected for processing (13)
.github/workflows/pr-review-advisor.yamltest/e2e-recommendations.test.tstest/pr-e2e-gate.test.tstest/pr-review-advisor-comment-cli.test.tstest/pr-review-advisor-security-boundaries.test.tstest/pr-review-advisor-workflow-boundary.test.tstest/pr-review-advisor.test.tstools/advisors/e2e-recommendations.mtstools/advisors/github.mtstools/e2e/pr-e2e-gate.mtstools/pr-review-advisor/analyze.mtstools/pr-review-advisor/comment.mtstools/pr-review-advisor/workflow-boundary.mts
21 tasks
cv
added a commit
that referenced
this pull request
Jul 14, 2026
## Summary Separate trusted controller health from the exact-diff merge verdict so handled PR CI/E2E failures and superseded or closed revisions no longer masquerade as controller failures. Add self-explanatory diagnostics and exception links while preserving red controller status for real controller, correlation, and evidence-download faults. This follows the actual-run review of #6807 and #6810. ## Changes - Rename the workflow to `E2E / PR Gate Controller` while preserving the required custom check name `E2E / PR Gate`, and document which outcome controls merge authority. - Complete prerequisite-CI and selected-E2E failures as handled red PR verdicts with direct run, job, and failed-step links; paginate bounded workflow-job results and keep diagnostic lookup failures fail-closed at the PR check. - Cancel exact checks for superseded revisions and closed PRs, including deleted fork repositories, only after validating the trusted check identity. Bracket evidence parsing with live exact-diff reads so stale evidence cannot become a current result. - Keep infrastructure failures distinct: a successful child whose evidence download fails leaves both controller and custom check red, while classified missing, duplicate, skipped, pending, or reported-failing signals remain handled PR verdicts. - Make no-secret exceptions explicitly say that credentialed E2E did not run, embed the controller-run link needed for **Review deployments**, and clarify the exact manual evidence URL format. - Add lifecycle, pagination, workflow, command, exception, and race regression coverage; update the E2E reference and maintainer merge-gate guidance. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: No end-user product behavior changed; the internal E2E reference and maintainer merge-gate guidance were updated. - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Adversarial fail-closed review covered GitHub App/external-ID correlation, stale and deleted-fork revisions, bounded diagnostics, evidence trust boundaries, download failures, and pre/post-parse race validation; no remaining blockers. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run --project integration test/pr-e2e-gate-command.test.ts test/pr-e2e-gate.test.ts test/pr-e2e-gate-lifecycle.test.ts test/pr-e2e-gate-exceptions.test.ts test/pr-e2e-gate-workflow.test.ts` (5 files, 107 tests passed) - [x] Applicable broad gate passed — `GIT_CONFIG_GLOBAL=/dev/null SSH_AUTH_SOCK=/tmp/ssh-qmH9wX3sr5/agent.4170011 npm test` (1,502 files and 17,062 tests passed; 3 files and 40 tests skipped) - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Renamed the E2E gate workflow to **E2E / PR Gate Controller** with stronger handling for superseded, closed, cancelled, and timeout scenarios. * Improved evidence verification and finalization reporting, including evidence outcome propagation and clearer fail-closed behavior. * Updated wait/completion behavior so terminal failures/timeouts are reported consistently without unexpected non-zero exits, and improved check conclusion/link messaging (including **cancelled**). * **Documentation** * Refined controller/manual dispatch guidance, evidence handling rules, and “no-secret exception” workflow instructions. * **Tests** * Updated and expanded E2E lifecycle, workflow contract, and CLI parsing/evidence outcome coverage to match the new behavior and messages. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This follow-up to #6745 and #6788 makes the combined PR Review Advisor disclose both model lanes without giving Nemotron authority over GPT's assessment, fixes contradictory confidence and E2E guidance, and ensures stale CI events close only the seeded in-progress E2E check. Missing or malformed Nemotron artifacts now degrade to an unavailable second opinion while a valid GPT review still publishes.
Changes
github-actions[bot]comments whose first line is one of the two retired E2E Advisor markers, after the combined sticky comment publishes. The bot-owner and exact-marker restrictions are covered by the security-boundary tests.in_progress, allowing the existing cleanup step to terminalize abandoned checks without rewriting completed historical results.Type of Change
Quality Gates
pull_request_targetprivilege separation, same-run artifact provenance, primary fail-closed behavior, nonblocking secondary validation, prose isolation, exact bot-owned legacy comment deletion, E2E selector trust, and stale/completed check races. The nine-category security checklist also found no secret, injection, authorization, dependency, logging, cryptography, configuration, test, or holistic posture regression.Verification
Verifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailablenpm testfor broad runtime/test-harness changes;npm run checkfor repo-wide validation/coverage changes — command/result:npm run docsbuilds without warnings (doc changes only)Signed-off-by: Carlos Villela cvillela@nvidia.com
Summary by CodeRabbit