Skip to content

Raise vitest testTimeout to absorb per-file parser init#2822

Merged
NullVoxPopuli merged 1 commit into
ember-cli:masterfrom
NullVoxPopuli-ai-agent:investigate-test-flakiness
Jul 3, 2026
Merged

Raise vitest testTimeout to absorb per-file parser init#2822
NullVoxPopuli merged 1 commit into
ember-cli:masterfrom
NullVoxPopuli-ai-agent:investigate-test-flakiness

Conversation

@NullVoxPopuli-ai-agent

Copy link
Copy Markdown

Symptom

On a loaded machine, the suite fails 1–20+ random valid cases per run, a different set each time. The failure is always:

Error: Test timed out in 5000ms.

from inside ESLint's RuleTester — never an actual wrong assertion.

Mechanism

  1. Vitest isolates each test file (default), resetting the module registry — so the first RuleTester case in every file re-pays parser initialization: content-tag's wasm compile, glimmer, and the typescript-eslint machinery. Measured: the first test in a template-rule file runs 950–1560ms, every subsequent test 2–24ms, even when two files share a worker process.
  2. Vitest's default per-test timeout is 5000ms — only ~4× headroom over that first-test cost.
  3. Parallel forks + any external CPU load stretch that ~1s init past 5s, and vitest kills whichever file's first test loses the race — hence the random-looking failures. no-array-prototype-extensions (588ms first case, TS-heavy) is affected too, matching what actually fails in practice.

Verification

Reproduced deterministically on a 4-core machine: running the suite alongside six yes > /dev/null busy-loops fails template-no-unbound > valid > <template>{{foo}}</template> at exactly 5000ms. With testTimeout: 30_000 and the same load, all 9538 tests pass.

Alternative considered

Warming the parser in a setupFiles hook would shave the per-file cost instead of tolerating it, but the init happens lazily inside the parser on first parse per module registry, so it would need to import and exercise the parser in every isolated context anyway — the timeout bump is the simpler, standard remedy and doesn't change what the tests exercise.

🤖 Generated with Claude Code

Vitest isolates each test file, so the first RuleTester case in every
file re-pays ~1-1.5s of ember-eslint-parser initialization (content-tag
wasm compile, typescript-eslint machinery). Under CPU contention that
first case can exceed the default 5s per-test timeout, failing a random
valid case per run — the suite is flaky on loaded machines. Reproduced
deterministically by running the suite alongside six busy-loops on a
4-core box; with a 30s timeout the same load passes all 9538 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@NullVoxPopuli NullVoxPopuli merged commit 24526b3 into ember-cli:master Jul 3, 2026
12 checks passed
@NullVoxPopuli NullVoxPopuli deleted the investigate-test-flakiness branch July 3, 2026 19:35
@github-actions github-actions Bot mentioned this pull request Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants