Skip to content

test: give every scaling sample a clock-tick floor - #50

Merged
quantizor merged 3 commits into
mainfrom
test/scaling-sample-floor
Nov 19, 2026
Merged

quantizor merged 3 commits into
mainfrom
test/scaling-sample-floor

Conversation

@quantizor

@quantizor quantizor commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

The scaling check failed the Release workflow on main for a linear operation: substitution (selector placeholders whose rule body opens on the next line): N=6000 0.997ms, 4N=24000 6.996ms, ratio=7.02 (threshold 7).

GitHub's ubuntu runners read thread CPU time in whole-millisecond ticks. A check whose N run takes 1 to 2 ms keeps a minimum sample rounded down to one tick, which inflates its ratio past the threshold. Enlarging four specific cases fixed those cases only.

Each sample now repeats the operation back to back until it spans a 20 ms CPU floor. The repetition count is calibrated once per check at N, doubling until the smaller of two samples clears the floor, and is reused at 4N. The printed time is per run, so one clock tick is at most about 5% of any sample, whatever the check costs. Each output line also prints the runs per sample and the CPU time per N sample.

Each timed run of a language feature now gets a new template context, the way tsserver brings one per request. Before, every run reused one context object, so the plugin's per-context caches (the line map and the escape-replaced text) were built during warmup and never timed: a quadratic injected into line-map building left the diagnostics check at 3.51. With a new context per run, the same injection reads about 7.

Two comment substitution cases keep a larger N. On GitHub's runners their per-character cost steps up between 6,000 and 24,000 units, so they read 5.9 to 6.5 at N=6,000 across three CI runs. At N=24,000 both sizes are past the step. The cause of the step is unconfirmed, and the rise levels off at larger sizes rather than compounding, which a hidden quadratic would do.

Testing

  • corepack yarn test:scaling passed twice in a row on a loaded machine, every ratio between 2.85 and 5.11.
  • With the CPU clock floored to whole milliseconds to simulate the runners, the full run passes.
  • A quadratic scan injected into getTemplateSubstitutions still fails the gate on every attempt (ratios 9.2 to 12.9 for a light injection).
  • A scaling run takes about 2.3 times its previous wall time, roughly 20 seconds more locally.

@changeset-bot

changeset-bot Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 1780f27

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound — a calibrated repetition count applies the clock-tick floor to every check instead of enlarging a few fixtures, and the same count cancels out of the 4N/N ratio.

Reviewed changes in this run: the full diff (2 files, 1 commit), plus the surrounding timing path, fixture caching, and every ScalingCheck definition.

  • Sample floor — measure now runs the operation repetitions times and reports per-run CPU time; calibrateRepetitions doubles at N until the sample spans 20ms, then reuses the count at 4N.
  • Shared N restored — the per-case n override is gone, so all substitution and escape cases run at SUBSTITUTION_CHECK_N again.
  • Docs — docs/maintenance.md now describes samples, the floor, and per-run timing.

I ran corepack yarn test:scaling on this branch: every check passed with no retries, including the previously failing selector case at ratio 4.45. I also checked that each repetition creates a fresh language service and that reused fixtures are warmed before timing, so verifying only the last result stays meaningful.

Pullfrog  | View workflow run | Using big-pickle (free) | 𝕏

On GitHub's runners the two comment substitution cases read near 6 at
N=6000 across three CI runs, while every per-character cost is flat
locally from 1,500 to 384,000 units: their 4N text outgrows a CPU cache
that the N text fits. They run at N=24,000, past it.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes since the prior review (441e2b7): one commit reintroduces targeted N overrides for the two cheapest-per-character cases.

  • Restored the n override plumbing — TextCheckCase.n and defineTextCheck's optional n = SUBSTITUTION_CHECK_N are back, and substitutionCases.map passes each case's n through to defineCheck.
  • Re-scoped the override constant — COARSE_CLOCK_TEXT_CHECK_N became CACHE_BOUND_TEXT_CHECK_N (still 24,000) with a comment explaining the CPU-cache rationale; it now applies only to the two block-comment substitution cases, while the escape and adjacent-placeholder cases stay at the shared N.
  • Validated the guardrail — corepack yarn test:scaling passed every check with no retries (both comment cases read 4.21 and 4.01 at N=24,000) and corepack yarn typecheck is clean.

Pullfrog  | View workflow run | Using big-pickle (free) | 𝕏

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes since the prior review (5aa279c): one commit gives each timed language-feature run a fresh template context and tightens the sample calibration.

  • Fresh context per run — freshContextForSize caches a context factory per size (keeping the source-file parse cached) and calls it per use, so diagnostics, folding, hover, completions, and code-fix checks no longer carry per-context memos between runs. Verification helpers that fetch a context again now get a fresh one, still outside the timed region.
  • Two-sample calibration — calibrateRepetitions requires the smaller of two samples at a count to clear the 20 ms floor before stopping, avoiding an early stop on a single noise-inflated sample when the timed minimum would sit below the floor.
  • Clearer output and prose — every line now prints runs per sample and CPU ms per N sample; docs/maintenance.md describes both changes, and the cache-step comment no longer claims a confirmed cause.

Ran corepack yarn test:scaling locally: all 36 checks passed with no retries, ratios 2.70 to 4.53. corepack yarn typecheck is clean.

Pullfrog  | View workflow run | Using big-pickle (free) | 𝕏

@quantizor
quantizor merged commit 20c5320 into main Nov 19, 2026
12 checks passed
@quantizor
quantizor deleted the test/scaling-sample-floor branch November 19, 2026 01:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant