Skip to content

fix(ci): stabilize CLI coverage shard ownership#6747

Merged
cv merged 3 commits into
mainfrom
codex/fix-coverage-shard-stability
Jul 13, 2026
Merged

fix(ci): stabilize CLI coverage shard ownership#6747
cv merged 3 commits into
mainfrom
codex/fix-coverage-shard-stability

Conversation

@cv

@cv cv commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Replace roster-sensitive CLI coverage allocation with stable project/path ownership so adding, removing, or renaming one test no longer moves unrelated tests between coverage shards. The eight-shard parallel run remains intact, and the checked-in timing model keeps the slowest shard within 5% of average load.

The first coverage comparison for this PR may show one final set of per-file changes because it intentionally moves from the old allocation to the stable baseline; subsequent test-roster changes will preserve existing ownership.

Changes

  • Assign each CLI and integration test to a shard with a fixed SHA-256 hash of its project and repository-relative path.
  • Keep timing hints as an estimated-load measurement and fail a deterministic 1,344-entry balance fixture if its slowest shard exceeds 105% of average load. The actual current roster has a 933,721 ms estimated maximum versus the previous allocator's 898,605 ms maximum (+3.9%). Across three live CI runs, the slowest shard's wall time was 5m50s, 5m51s, and 6m14s; the final run's test command took 5m26s versus 5m22s on PR ci(advisors): consolidate PR and E2E guidance #6745 (+1.3%), while total job wall time was 6m14s versus 6m08s (+1.6%).
  • Add regression coverage proving insertions and removals do not move existing files, discovery order does not matter, fixed ownership remains recorded, and all files are assigned exactly once.
  • Record the escaped-defect boundary in the tests: the prior longest-processing-time allocator was deterministic for one roster but recomputed every ownership when that roster changed. Existing tests covered deterministic discovery order and balance, but not ownership stability across roster changes.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: internal CI/test-harness partitioning only; contributor documentation remains accurate
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification:
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project integration test/cli-coverage-sequencer.test.ts (8 tests passed)
  • Applicable broad gate passed — npm test (1,468 files and 16,584 tests passed; 3 files and 40 tests skipped)
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

@cv cv self-assigned this Jul 13, 2026
@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

CLI coverage sharding now assigns tests deterministically by project and path using salted hashing. The sequencer applies this behavior to CLI and integration projects, while tests verify stable ownership, roster balance, recorded assignments, and project gating.

Changes

Stable CLI coverage sharding

Layer / File(s) Summary
Deterministic shard assignment
test/helpers/cli-coverage-sequencer.ts
Adds salted SHA-256 shard assignment keyed by entry identity and replaces duration-aware project gating with CLI coverage project gating.
Sequencer integration and stability validation
test/helpers/cli-coverage-sequencer.ts, test/cli-coverage-sequencer.test.ts
The sequencer keys assignments by project and path, while tests construct representative rosters and verify stable ownership, recorded assignments, balance, and project selection.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: stabilizing CLI coverage shard ownership in CI.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-coverage-shard-stability

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage remains at 96%, unchanged from the branch.

TypeScript / code-coverage/cli

The overall coverage in the branch remains at 79%, unchanged from the branch.

Show a code coverage summary of the most impacted files.
File deed1aa 1c35156 +/-
src/lib/actions.../status-text.ts 88% 43% -45%
src/lib/actions...up-rendering.ts 78% 34% -44%
src/lib/actions...e-classifier.ts 86% 48% -38%
src/lib/cli/public-dispatch.ts 85% 69% -16%
src/lib/shields/index.ts 70% 62% -8%
src/lib/sandbox/config.ts 73% 66% -7%
src/lib/state/sandbox.ts 76% 85% +9%
src/lib/inferen...ollama/proxy.ts 26% 35% +9%
src/lib/state/o...config-merge.ts 84% 96% +12%
src/lib/onboard...er-gpu-patch.ts 73% 88% +15%

Updated July 13, 2026 06:31 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: None

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • None. No E2E is needed: this tests-only CI tooling change affects how existing CLI/integration unit tests are assigned to coverage shards, with dedicated unit coverage for stable assignment, roster changes, balance, and sequencer wiring. It cannot affect a real NemoClaw user or runtime flow.

Optional E2E

  • None.

New E2E recommendations

  • None.

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: No advisor follow-up required beyond maintainer review.
Findings: 0 required · 0 warnings · 0 optional suggestions
Since last review: 0 prior items resolved · 0 still apply · 0 new items found

Workflow run details

This is an automated review. Required findings need action before merge. Warnings and optional suggestions do not require a response or follow-up. A human maintainer makes the final merge decision.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/cli-coverage-sequencer.test.ts`:
- Line 4: Replace the live repository lookup through
checkedInCliCoverageEntries() in the CLI coverage sequencer test with a fixed
roster fixture, keeping the test deterministic and independent of the Git
worktree and executable. If repository validation is still needed, move it into
a separate integration test.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1924c424-b14d-4f24-98a9-d4aafc1cb23a

📥 Commits

Reviewing files that changed from the base of the PR and between deed1aa and 27d76ec.

📒 Files selected for processing (2)
  • test/cli-coverage-sequencer.test.ts
  • test/helpers/cli-coverage-sequencer.ts

Comment thread test/cli-coverage-sequencer.test.ts Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/cli-coverage-sequencer.test.ts`:
- Around line 131-136: Update the assertion in the “keeps a representative test
roster balanced across the eight CI shards” test to enforce a maximum shard load
of 105% of average by changing the 1.06 multiplier to 1.05. Preserve the
existing shard assignment and weight calculations.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d903a799-4edc-4e29-9311-2be58371e1de

📥 Commits

Reviewing files that changed from the base of the PR and between 27d76ec and e960b76.

📒 Files selected for processing (1)
  • test/cli-coverage-sequencer.test.ts

Comment thread test/cli-coverage-sequencer.test.ts Outdated
@cv cv merged commit ddf951a into main Jul 13, 2026
59 of 62 checks passed
@cv cv deleted the codex/fix-coverage-shard-stability branch July 13, 2026 07:43
@github-actions github-actions Bot added the v0.0.82 Release target label Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v0.0.82 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants