Repository navigation
test: give every scaling sample a clock-tick floor - #50
Conversation
|
There was a problem hiding this comment.
✅ No new issues found.
Approach: sound — a calibrated repetition count applies the clock-tick floor to every check instead of enlarging a few fixtures, and the same count cancels out of the 4N/N ratio.
Reviewed changes in this run: the full diff (2 files, 1 commit), plus the surrounding timing path, fixture caching, and every ScalingCheck definition.
- Sample floor —
measurenow runs the operationrepetitionstimes and reports per-run CPU time;calibrateRepetitionsdoubles at N until the sample spans 20ms, then reuses the count at 4N. - Shared N restored — the per-case
noverride is gone, so all substitution and escape cases run atSUBSTITUTION_CHECK_Nagain. - Docs —
docs/maintenance.mdnow describes samples, the floor, and per-run timing.
I ran corepack yarn test:scaling on this branch: every check passed with no retries, including the previously failing selector case at ratio 4.45. I also checked that each repetition creates a fresh language service and that reused fixtures are warmed before timing, so verifying only the last result stays meaningful.
big-pickle (free) | 𝕏
On GitHub's runners the two comment substitution cases read near 6 at N=6000 across three CI runs, while every per-character cost is flat locally from 1,500 to 384,000 units: their 4N text outgrows a CPU cache that the N text fits. They run at N=24,000, past it.
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes since the prior review (441e2b7): one commit reintroduces targeted N overrides for the two cheapest-per-character cases.
- Restored the
noverride plumbing —TextCheckCase.nanddefineTextCheck's optionaln = SUBSTITUTION_CHECK_Nare back, andsubstitutionCases.mappasses each case'snthrough todefineCheck. - Re-scoped the override constant —
COARSE_CLOCK_TEXT_CHECK_NbecameCACHE_BOUND_TEXT_CHECK_N(still 24,000) with a comment explaining the CPU-cache rationale; it now applies only to the two block-comment substitution cases, while the escape and adjacent-placeholder cases stay at the shared N. - Validated the guardrail —
corepack yarn test:scalingpassed every check with no retries (both comment cases read 4.21 and 4.01 at N=24,000) andcorepack yarn typecheckis clean.
big-pickle (free) | 𝕏
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes since the prior review (5aa279c): one commit gives each timed language-feature run a fresh template context and tightens the sample calibration.
- Fresh context per run —
freshContextForSizecaches a context factory per size (keeping the source-file parse cached) and calls it per use, so diagnostics, folding, hover, completions, and code-fix checks no longer carry per-context memos between runs. Verification helpers that fetch a context again now get a fresh one, still outside the timed region. - Two-sample calibration —
calibrateRepetitionsrequires the smaller of two samples at a count to clear the 20 ms floor before stopping, avoiding an early stop on a single noise-inflated sample when the timed minimum would sit below the floor. - Clearer output and prose — every line now prints runs per sample and CPU ms per N sample;
docs/maintenance.mddescribes both changes, and the cache-step comment no longer claims a confirmed cause.
Ran corepack yarn test:scaling locally: all 36 checks passed with no retries, ratios 2.70 to 4.53. corepack yarn typecheck is clean.
big-pickle (free) | 𝕏

The scaling check failed the Release workflow on
mainfor a linear operation:substitution (selector placeholders whose rule body opens on the next line): N=6000 0.997ms, 4N=24000 6.996ms, ratio=7.02 (threshold 7).GitHub's ubuntu runners read thread CPU time in whole-millisecond ticks. A check whose N run takes 1 to 2 ms keeps a minimum sample rounded down to one tick, which inflates its ratio past the threshold. Enlarging four specific cases fixed those cases only.
Each sample now repeats the operation back to back until it spans a 20 ms CPU floor. The repetition count is calibrated once per check at N, doubling until the smaller of two samples clears the floor, and is reused at 4N. The printed time is per run, so one clock tick is at most about 5% of any sample, whatever the check costs. Each output line also prints the runs per sample and the CPU time per N sample.
Each timed run of a language feature now gets a new template context, the way tsserver brings one per request. Before, every run reused one context object, so the plugin's per-context caches (the line map and the escape-replaced text) were built during warmup and never timed: a quadratic injected into line-map building left the diagnostics check at 3.51. With a new context per run, the same injection reads about 7.
Two comment substitution cases keep a larger N. On GitHub's runners their per-character cost steps up between 6,000 and 24,000 units, so they read 5.9 to 6.5 at N=6,000 across three CI runs. At N=24,000 both sizes are past the step. The cause of the step is unconfirmed, and the rise levels off at larger sizes rather than compounding, which a hidden quadratic would do.
Testing
corepack yarn test:scalingpassed twice in a row on a loaded machine, every ratio between 2.85 and 5.11.getTemplateSubstitutionsstill fails the gate on every attempt (ratios 9.2 to 12.9 for a light injection).