e2e analyzer: let the rubric reach a test-side verdict instead of defaulting to product - #15340
Merged
Merged
Conversation
Three consecutive merge-lane runs (30944006762, 30948032053, 30952155263)
put the same two new-folder-flow renv failures on "suspected product
regression" when the mechanism was visible in the test's own helper. Both
reports even recommended the right test-side fix ("wait for renv install
to be confirmed") while labeling the root cause product -- the rubric's
residual rule has no test-side exit, so an unexplained failure can only
land on product.
- Say plainly that `test logic bug` and `locator drift` are not
dismissals, so a cited test-side mechanism outranks the residual, and
add the self-check: recommending a test fix while labeling the cause
product contradicts itself.
- New section on verifying the test established its own precondition:
output that marks the START of an async operation proves initiation, not
completion, and a readiness check returning in tens of milliseconds
proves nothing. Time the setup step from the trace timestamps.
- New subsection under the clock rules for where a wait BEGAN: subtract an
intervening reload from the budget before calling a product open path
slow. 23s of a 40s wait spent reloading is an under-budgeted wait, not a
slow handler.
- Note that a "first seen" date is censored by the lookback window, so it
is only an onset when clean runs precede it inside the window.
Validating the previous commit against the three runs it was written for fixed both ubuntu reports (they now name the install-completion race in packageManager.ts and say to fix the helper, not the timeout) but pushed the Windows report from "suspected product regression" to "locator drift," recommending a change to a selector that is correct: `.simple-title-bar` is the parent of `.simple-title-bar-title` in draggableTitleBar.tsx:103, so the existing filter matches. - Scope the "not a dismissal" promotion to `test logic bug`. Locator drift keeps its own stricter decision procedure; permission to reach a test-side verdict is not permission to blame the selector. - Add the determinism guard the rubric was missing: markup does not revert between runs, so a selector that mostly passes cannot be stale. A high per-environment pass rate refutes drift outright, and a failure-state screenshot showing the target rendered with the filtered text means the question is timing, not markup. - Route "present with expected role, visibility timeout" through the wait-window subsection before it lands on `flaky test`, since an under-budgeted wait is a test logic bug with a specific fix.
…nsored
The censoring rule added in the previous commits was applied inconsistently
across the validation runs: the payload carries `lookback_days` but no date
for where the window opens, so using it required the model to do the
subtraction, and it usually skipped it and reported "first seen Jul 28"
bare.
Compute the boundary in e2e-query-history.js and emit it as `window_start`
(listed first so it reads next to `lookback_days` and survives any
head-keeping truncation of the payload). Prefer the API's own
`lookback_days`, since it may clamp what we asked for, and let an
API-provided `window_start` win if the endpoint ever grows one. The error
paths still print a bare `{}`, which the rubric reads as "API unreachable."
Rubric and the script README now name the field instead of describing the
arithmetic.
|
E2E Tests 🚀 Note No feature tags detected. If this PR needs feature coverage, add the tag above and retrigger the workflow. |
jonvanausdeln
enabled auto-merge (squash)
August 4, 2026 23:55
midleman
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three consecutive merge-lane runs put the same two new-folder-flow renv failures on "suspected product regression" when the mechanism was visible in the test's own helper. Both reports even recommended the right test-side fix while labeling the root cause product, which routes the work to the wrong owner. The analyzer had all the evidence it needed in both cases, so this is a reasoning-rubric change, not a data change.
Summary
The hole: the residual rule in
rubric.mdnames four dismissive categories (flaky, timeout, infrastructure, test environment) and says that when no dismissal is earned and it isn't locator drift, the residual is a suspected product regression.test logic bugnever appears in that path, so an unexplained failure can only land on product.Four changes:
test logic bugis not a dismissal -- it names an owner and a fix, so a cited test-side mechanism outranks the residual. Adds the self-check that recommending "harden the test to wait for X" while labeling the cause product contradicts itself. Explicitly does not extend to locator drift, which keeps its own stricter decision procedure.[before]/[after]stamps. Notes that this failure mode is intermittent by nature, so a high pass rate does not refute it.[after deadline]lines). Subtract an intervening window reload from the budget before calling a product open path slow: 23s of a 40s wait spent reloading is an under-budgeted wait, not a slow handler.window_startin the history payload, plus the rule that a "first seen" date at the window's edge means the onset is unknown rather than established.rubric.mdis injected verbatim into the Action's system prompt and shared with the interactivee2e-failure-analyzerskill, so both consumers pick this up.Validation
Dispatched the analyzer from this branch against the three runs the changes were written for, and read every report.
packageManager.ts, cites the ~5s trace timing, "the modal is expected product behavior"Both diagnoses match the fixes in #15338, which were derived independently by hand.
An intermediate revision is worth calling out, because it is the risk this kind of change carries: the first version promoted
locator driftalongsidetest logic bug, and the Windows report then swung to "stale selector," recommending a change to a selector that is correct (.simple-title-baris the parent of.simple-title-bar-titleindraggableTitleBar.tsx:103, so the existinghasTextfilter matches). The report even called the modal "fully rendered and visible" while claiming its selector did not match. Commit 2 scopes the promotion and adds the guard that was missing: locator drift is deterministic, so a selector that passes 99% of the time cannot be stale, and a high per-environment pass rate refutes drift outright.Release Notes
New Features
Bug Fixes
Validation Steps
CI tooling only -- no product or e2e-test code changes, so no test tags apply. Validation is the three dispatched analyzer runs above; the reports are attached to each as the
e2e-failure-analysisartifact. To re-check any run: Actions -> "Analyze E2E Failures" -> Run workflow, pick this branch, paste a run URL.