[perf][regression] Large minified bundle becomes a multi-hour compile after RS4GC/opt-tier changes (83 CGUs, ~17 GiB RSS)

[proggeramlug]

What happened

A cold native compile of the public @anthropic-ai/claude-code@2.1.112
standalone bundle no longer finishes in the previously observed time on current
Perry main.

On Perry 0.5.1515 (b00e261f8114dd0f39d061131703a918e17b408f), the
compiler selected 83 codegen units, spent about 27 minutes partitioning the
already-lowered module, reached approximately 17 GiB RSS, and then took 11--14
minutes for each of the first LLVM units that completed. I interrupted the run
after 74.5 minutes because only two units had completed and the remaining
linear ETA was multiple hours.

No output binary was produced, so this also prevents measuring the intended
runtime/GC performance of current Perry on this real application.

This appears to be a compile-time regression caused by the interaction of:

1. native roots/RS4GC becoming the default;
2. removal of the post-RS4GC instruction-count guard and the older oversized
   function/unit -O0 fallback;
3. -Os becoming the default for every native module; and
4. IR-size-based partitioning increasing this input from approximately 6 to 83
   codegen units without being able to split the largest individual functions.

The important point is that inter-function codegen-unit splitting cannot bound
LLVM work for one giant minified IIFE. Perry's own source comment identifies a
68 MB IIFE in this exact cli.js case as irreducible without structured
intra-function outlining.

What I expected

The default compile should finish in a bounded amount of time and memory,
without sending RS4GC-expanded multi-million-instruction functions through an
unbounded optimizing middle-end.

As a historical reference, the same 13.7 MB input compiled on Perry commit
4340bffc9bb7500aaf1863a3ba334921c3a92fe8 (2026-07-29) in approximately
40--50 minutes with roughly 5 GiB peak RSS. That build produced a working
native executable.

The current backend and GC strategy do not need to reproduce the old wall time
exactly, but a default compile should complete rather than regress toward a
multi-hour build with more than 3x the recorded memory use.

Reproduction

This is a scale-triggered compiler issue, so I have not reduced it to a small
TypeScript snippet. The input is a deterministic, publicly obtainable npm
artifact.

1. Build current Perry and the runtime libraries

git clone https://github.com/PerryTS/perry.git
cd perry
git checkout b00e261f8114dd0f39d061131703a918e17b408f

cargo build --release -p perry
cargo build --release \
  -p perry-runtime-static \
  -p perry-stdlib-static \
  -p perry-wasm-host \
  --features perry-runtime/wasm-host

export PERRY_RUNTIME_DIR="$PWD/target/release"

2. Obtain the standalone Claude Code bundle

workdir="$(mktemp -d)"
cd "$workdir"
npm pack @anthropic-ai/claude-code@2.1.112
tar -xzf anthropic-ai-claude-code-2.1.112.tgz

wc -c package/cli.js
shasum -a 256 package/cli.js

Expected input identity:

13711684 package/cli.js
bc3358282800e3e99daa8e71ac5b7b1566bd0d7ca7eb94f714a7859365d3163f  package/cli.js

Version 2.1.112 is used because it is the last Claude Code package in this
series that contains the standalone cli.js bundle; later packages are thin
download shims and are not equivalent compiler inputs.

3. Run a cold compile with codegen diagnostics

export PERRY_CODEGEN_PROGRESS=all
export PERRY_CODEGEN_UNIT_TIMINGS=1

/path/to/perry/target/release/perry compile \
  --no-auto-optimize \
  --no-cache \
  --enable-wasm-runtime \
  -v \
  package/cli.js \
  -o ./cc

--no-auto-optimize makes this a cleaner application-codegen reproduction and
matches the successful July command. The observed current run omitted that
flag, but the stall was inside application LLVM codegen, before the
runtime/stdlib auto-optimization stage, so it is not the source of this issue.

For RSS on macOS, run the command through /usr/bin/time -l, or sample the
perry process while it is in the partitioning/freezing/LLVM codegen phase.

Observed diagnostic output/timeline

The current compiler reported:

[perry] codegen: lowering ... (35501 callables)
...
[perry] codegen: ...: partitioning into 83 codegen units
...
[perry] codegen: ...: freezing 83 codegen units for worker threads
[perry] codegen: ...: freeze/LLVM pipeline started: 83 units, 2 workers, ~1773.1 MiB estimated IR
...
[perry] codegen: ...: LLVM unit 2/83 finished (808.3s; ...)
[perry] codegen: ...: LLVM unit 3/83 finished (660.7s; ...)

Approximate phase timings from the cold default run:

Phase	Observation
HIR/codegen lowering before partition announcement	~12.3 min
Partition construction before freezing 83...	~27 min
First completed LLVM units	660.7 s and 808.3 s
Peak compiler RSS observed	17,472,160 KiB (~16.7 GiB)
Run stopped	74.5 min, no binary

The callable-count policy alone would choose
ceil(35501 / 6000) = 6 units. Therefore the 83-unit decision comes from the
new 20 MiB estimated-function-IR arm and implies roughly 1.6 GiB of function IR
before RS4GC runs. The reported 1773.1 MiB includes the per-unit scaffold as
well. RS4GC can then expand individual functions further inside each LLVM
worker, which is not represented by that pre-rewrite estimate.

Two additional exploratory arms did not improve the result:

• PERRY_LLVM_INPROCESS=0: still partitioning/materializing the 83 textual
  units after 65 minutes; clang workers had not started.
• PERRY_CODEGEN_UNITS=1 PERRY_LLVM_INPROCESS=0: still finalizing generated IR
  after 50 minutes.

Those two exploratory runs, especially the single-unit arm, were affected by
other Rust/Perry builds running on the same host and should not be treated as
clean comparative benchmarks. The primary run's absolute wall time was also
hurt by host contention and low free disk. However, contention does not explain
the deterministic policy changes, the 83-unit selection, the 27-minute
partition construction, or the known RS4GC giant-function behavior described
below.

Environment

• Perry version: perry 0.5.1515
• Perry commit: b00e261f8114dd0f39d061131703a918e17b408f
• Host OS: macOS 26.5 (25F71), arm64
• CPU: Apple M1 Max, 10 logical CPUs
• Memory: 64 GiB
• Target: native arm64-apple-darwin
• Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
• Cargo: cargo 1.97.1 (c980f4866 2026-06-30)
• Apple clang: 21.0.0 (clang-2100.1.1.101)
• Installed via: built from source in a clean detached worktree

Suspected regression and source evidence

1. RS4GC became the default after the successful baseline

Commit 148f97bc2
(#7370, 2026-08-04) made native roots/statepoints the default on supported
targets. The July baseline predates that change and used the shadow-stack
lowering unless RS4GC was explicitly enabled.

This matters because Perry has already measured pathological RS4GC relocation
fan-out on minified bundles. Commit
e34b84029
(#8128) records one 51k-line closure growing 40x to 2.1 million instructions;
one -Os function pass then ran for more than 65 CPU-minutes without
finishing.

2. The guard for that exact blow-up was removed

#8128 added a post-RS4GC guard: functions above 512k instructions were stamped
optnone + noinline before the normal optimization pipeline. Its commit
message reports that the affected unit then completed in approximately 21
seconds.

Commit 9051d3fd2
(#8421, 2026-08-19) removed that post-rewrite guard. The same commit also
removed the older oversized-module and widest-function -O0 routing from
linker.rs.

The current replacement, PERRY_LL_PREOPT_OPTNONE_INSTRS, is opt-in and has a
default of zero. It is also evaluated before
rewrite-statepoints-for-gc. Consequently, it cannot detect a function that
is moderate before RS4GC but explodes only after relocation fan-out unless the
user guesses a sufficiently low pre-rewrite threshold.

3. -Os is now unconditional by default

Commit 40e2ce86d
(#8457, 2026-08-20) made -Os the default native optimization level. Its source
comment explicitly records an approximately 21% compile-time cost on the
bundle used for that decision.

On current main, PERRY_LL_SIZE_OPT=0 changes -Os to -O3; it no longer
restores the historical pathological-function -O0 escape hatch. Both -Os
and -O3 run an optimizing middle-end and can be super-linear for the giant
function shape that the removed guard was designed to catch.

4. Auto-partitioning now chooses 83 units but cannot split the bottleneck

Commit 81a88de40
(#8017, 2026-08-13) changed decide_codegen_units from callable-count-only
sizing to the maximum of callable count and estimated function IR, targeting
20 MiB per unit with a cap of 128.

That policy selects 83 units here instead of the approximately 6 units selected
by the July formula. The partitioner correctly balances functions largest
first, but its own comment says the 68 MB Claude IIFE is irreducible through
inter-function splitting. Therefore more units add partitioning, scaffolding,
partial-link, and scheduling overhead without bounding LLVM's work on the
largest bodies.

The native path also uses two codegen-unit workers by default. With 83 units
and early unit times already around 11--14 minutes, bounded concurrency does
not make the default build practical.

Why this does not look like only machine contention

The host was not quiet, so the exact wall-clock values should be remeasured on
an idle machine. Nevertheless, the suspected mechanism is independently
documented in Perry's own history:

• the same RS4GC 40x relocation growth was previously reproduced;
• a post-rewrite guard was added specifically because an -Os pass exceeded
  65 minutes;
• that guard reduced the affected unit to about 21 seconds;
• the guard and the giant-function -O0 fallback are absent from current
  main;
• current main now sends all functions through -Os by default; and
• codegen-unit splitting explicitly cannot divide the largest Claude function.

The observed 660.7 s and 808.3 s unit completions are consistent with that
known failure mode even if contention inflated them.

Resolution direction

Restoring the #8128 post-RS4GC optnone guard is not the intended fix:
#8421 removed it on a principle that still stands (the optimization level is a
contract; compile-time scalability belongs in codegen, never in a silent
per-function opt downgrade). #8128's observation also still stands (statepoint
relocation is O(live roots x safepoints) and some shapes are not optimizable
after RS4GC expands them). Both describe the same function; what makes it
unoptimizable is the relocation fan-out, not its size.

The reconciled design therefore treats which precise-root lowering a function
uses as the per-function degree of freedom, not how hard LLVM optimizes it:

1. Make the native-root text lowering linear. function/precise_roots.rs
   (lower_roots_for_rs4gc, retype_landing_pads_for_statepoints) is
   O(lines x root-pointers) / O(pads x lines) and runs twice per function
   (partition reference scan + freeze). On this bundle that is the ~27-minute
   "partition" phase, and it never ran under the July shadow-stack default.
2. Delete the pre-RS4GC PERRY_LL_PREOPT_OPTNONE_INSTRS knob. optnone makes
   the pass manager skip the mem2reg/sccp prefix of the statepoint
   pipeline while RS4GC (a module pass keyed on the gc attribute) still
   runs, so a demoted function keeps root allocas the collector never sees.
   Default cap is 0 so nothing hits it today.
3. Replace the guard's safety role with a loud post-RS4GC instruction-budget
   assertion that fails the unit with the function name and pre/post counts
   instead of demoting. Bounded without touching the optimization contract.
4. Relocation-pressure root lowering ("root spilling"): at the sites that
   size a function's frame, estimate relocations (slots x call sites) and,
   above a threshold, keep that function's roots in a shadow frame instead of
   statepoints. The runtime already walks stack-map and shadow-frame roots in
   one visitor and the fp-chain walker skips record-less frames, and codegen
   emits the same body lines under both lowerings, so no new lowering or
   runtime machinery is needed. Every function keeps -Os; the choice is
   reported at default verbosity and in --report-size, with a strict mode
   that makes a spill an error.
5. Unit-count/worker policy from measured data (the 20 MiB target cannot be
   smaller than the largest indivisible function; 2 hard-coded native workers
   vs the text path's cores/4).
6. Structured outlining of the module-entry body remains the long-term shape
   fix (separate tracker).

Suggested acceptance criteria

• The default command above completes and produces a runnable binary on the
  public 2.1.112 bundle.
• No individual RS4GC-expanded function can spend unbounded time in -Os or
  -O3; the protective path is covered by a discriminating regression test.
• Codegen progress reports enough information to identify the widest
  pre-/post-RS4GC function and the selected optimization treatment.
• Peak memory and partition time do not scale pathologically merely because
  the IR-size heuristic selects dozens of codegen units.
• A shadow-root or no-middle-end bisection remains available, but the ordinary
  default build does not require undocumented environment overrides to finish.

[proggeramlug]

Investigation + fix series (measured)

Root-caused on origin/main and split into four stacked PRs. The reconciliation of #8421 (never silently change a function's optimization level) and #8128 (RS4GC relocation fan-out makes some functions unoptimizable) is in the issue body above; the short version is that the degree of freedom is which precise-root lowering a function uses, not how hard LLVM optimizes it.

What the code showed beyond the report

• The ~27-minute "partitioning" phase is a quadratic text pass, not partition bookkeeping. function/precise_roots.rs (lower_roots_for_rs4gc, retype_landing_pads_for_statepoints) is O(lines × root-pointers) / O(pads × lines) and runs on every stack-map function at render time, and the partitioner renders each function twice. It never ran under the July shadow-stack default — it arrived with native roots.
• The surviving PERRY_LL_PREOPT_OPTNONE_INSTRS knob is GC-unsound if enabled: optnone makes the pass manager skip mem2reg/sccp while RS4GC still runs, so the demoted function's root allocas are never promoted and the collector never sees them (proven by test).

Measurement (isolated @main unit, Homebrew LLVM 22.1.4)

The partitioner isolates the 68 MB IIFE (@main) into its own unit.

arm	result
@main pre-RS4GC	439,116 instrs; 795 root slots, ~106k call sites (HIR: 13,170 root slots × 5,622 safepoints)
RS4GC on the unit	62.0s, 7.1 GiB RSS; 439k → 6,532,814 instrs (~15×)
-Os on the un-rewritten unit	5.05s, 520 MiB — whole unit, @main included
-Os on the RS4GC'd 6.5M-instr unit (today)	did not finish under a 45-min cap

So the pathology is entirely RS4GC relocation fan-out on one function; remove the fan-out for it and -Os finishes in seconds.

The four PRs

• perf(codegen): make native-root text lowering linear in the function (#8583) #8585 — make precise_roots.rs linear (oracle-equivalence tested; 2000 roots × 44k lines: 49s → 0.29s). Fixes the 27-min partition phase.
• fix(codegen): delete the unsound pre-RS4GC optnone knob; add a post-rewrite instruction budget and per-unit codegen stats (#8583) #8586 — delete the unsound pre-RS4GC knob; add a post-rewrite per-function instruction budget (PERRY_LL_RS4GC_MAX_INSTRS, default 1.5Mi) that fails the unit loudly, never demotes; add per-unit codegen diagnostics under PERRY_CODEGEN_UNIT_TIMINGS.
• perf(codegen): spill giant-function GC roots to a shadow frame to bound RS4GC fan-out (#8583) #8589 (stacks on fix(codegen): delete the unsound pre-RS4GC optnone knob; add a post-rewrite instruction budget and per-unit codegen stats (#8583) #8586) — root spilling: a function whose estimate root_slots × safepoints exceeds PERRY_ROOT_SPILL_RELOCATIONS (default 4e6) keeps its roots in the shadow frame (pre-feat(gc): make native roots (statepoints) the default #7370 lowering) so RS4GC skips it. -Os and the runtime root walk are unchanged; the frame pointer is kept; each spill is reported. End-to-end mixed-frame GC differential test passes. Estimate distribution over all 83,310 functions: only @main at 74M, then a long tail (>4M: 1, >2M: 2, >1M: 2) — the default cleanly isolates @main.
• perf(codegen): CPU-aware default for native LLVM unit workers (#8583) #8593 (stacks on perf(codegen): spill giant-function GC roots to a shadow frame to bound RS4GC fan-out (#8583) #8589) — CPU-aware default for native LLVM unit workers (was a hard-coded 2; now half the CPUs clamped [2,8] on non-Windows, 2 on Windows).

Acceptance run (default threshold, on the real bundle)

The spill fires exactly as designed — @main's 13,170 roots go to a shadow frame — and the partition phase drops from ~27 min to ~9 min. The RS4GC fan-out regression is resolved.

It also surfaced a second, pre-existing pathology that #8421 removed a fallback for: dense generated closures (e.g. one 246k→491k-instr minified module factory with only ~130 roots — so not fan-out, ×1.6) cost rs4gc 121s + emit 272s per unit in -Os/SelectionDAG. That is the original #4880 "one giant generated function is slow at -O1+" case, whose -O0 (FastISel) routing #8421 also removed. It is orthogonal to GC roots (spilling doesn't reduce ISel), and there are many such closures across the ~84 units. #8593 (more workers) divides that floor so the compile completes in bounded time; it does not remove it.

Restoring the pre-regression ~40–50 min wall needs the #4880 fix too — structured intra-function outlining of the giant minified closures (the honest fix; module.rs's own comment already flags the 68 MB IIFE as irreducible without it), or a reported per-function FastISel routing for dense low-root functions. That is a separate, larger piece; filing it as a follow-up rather than folding an opt-level change back into this series, per #8421's principle.

[proggeramlug]

Follow-ups filed for the two architectural fixes discussed (the root causes behind the residual cost that root-spilling + more workers only bound, not remove):

• codegen: structured intra-function outlining of oversized generated functions (module-entry IIFE) #8595 — codegen: structured intra-function outlining of the oversized module-entry function. The real fix for both the RS4GC fan-out and the Compile-time blowup on wide object literals (2100 keys ≈ 2 min, 3000 keys > 7 min) #4880 ISel/emit cost: stop producing one 68 MB function. Keeps native statepoint roots everywhere; once it lands, this issue's root-spilling should rarely fire.
• gc: reduce safepoint density (safepoint polling instead of every-call statepoints) #8596 — gc: reduce safepoint density (safepoint polling vs every-call statepoints). Shrinks the × safepoints factor in the fan-out for the whole codebase, not just the outlier. Larger GC-architecture change; filed as an option to track.