Skip to content

feat: show what really broke behind failed tests in the PR comment - #4

Merged
bnsoni merged 1 commit into
feat/explain-resultsfrom
feat/show-causes
Oct 30, 2026
Merged

bnsoni merged 1 commit into
feat/explain-resultsfrom
feat/show-causes

Conversation

@bnsoni

@bnsoni bnsoni commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Stacked on #3. Needs iblai/iblai-deploy-ops#79 for the data; until then it renders exactly as #3.

Problem

Even with #3's headline, a developer with 61 failed tests still had to open the report, then a trace, then its Console tab, to find out that 37 of them were one crash, caused by the SDK bump in their own PR (os#617).

Change

When the central result carries causes (iblai/iblai-deploy-ops#79 groups failed tests by what really broke), the comment shows them under the headline:

Your build crashes in 37 of 61 failing tests: TypeError: Cannot read properties of undefined (reading 'find')

Tests What went wrong Example
37 App crash: TypeError: Cannot read properties of undefined (reading 'find') at 31836-4fad0d764232fb72.js:1:88062 non-admin goes to profile dropdown and opens the profile modal · screenshot · trace
13 Check failed: Error: expect(locator).toBeVisible() failed Locator: getByRole('button', { name: 'Open Accessibility Menu' }) user clicks the accessibility button… · screenshot · trace

Code at that spot:

…H=(0,p.m)({defaultValues:{facebook:k+(null!=(n=null==(r=null==M?void 0:M.social_links.find(e=>…

Hint: this PR changes @iblai/iblai-js 2.50.1 → 2.58.1. A crash right after a dependency bump usually comes from the new version.

  • Known failures go in a separate "Probably not from your PR" line, not in the table.
  • The hint appears only when a crash or browser error leads and the PR changes a dependency's version. The Publish step reads the PR's package.json diff with one compare API call using the caller's token; deps_from_patch keeps only changed versions.
  • Table safety: a pipe in error text can't break the table, and pipeline text still can't mention anyone.
  • Fix: a crash among failed tests (pattern: tests_failed_crash) is titled "❌ PR E2E failed", not "couldn't run your build".

Testing

  • tests/test-render.py, 11 new cases:
    • the table row with count, error, location and example links;
    • pipe escaping in cells;
    • the code quote;
    • the hint, and its two conditions;
    • known causes set apart;
    • the title fix;
    • dependency-diff parsing that counts only changed versions.
  • Mutation: 20 mutants on render.py, all caught.
  • The compare filter from action.yml (as YAML hands it to bash), run on Michael's SDK-bump commit f9bb8e0b: it yields @iblai/iblai-js 2.50.1 → 2.58.1.
  • Rendering: a full sample, rendered with gh api markdown, gives a table of 2 rows × 3 cells, the code block, and no stray pipes.

🤖 Generated with Claude Code

The central run now groups failed tests by their real cause (an app crash, a browser error, a
server error, or the check that failed), with an example test, its screenshot and trace, and the
code at a crash. The comment shows them as a table under the headline, quotes the code, keeps known
failures apart as "probably not from your PR", and, when a crash or browser error leads and the PR
changes a dependency's version in package.json, points at that bump. Table cells cannot be broken by
pipes in error text. A crash among failed tests is titled "failed", not "couldn't run your build".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bnsoni
bnsoni merged commit 302a1ec into feat/explain-results Oct 30, 2026
1 check passed
bnsoni added a commit that referenced this pull request Oct 30, 2026
Land #4 and #5 on main (they merged into their stacked parent branches)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant