Skip to content

perf(ci): skip LLVM AOT on the iOS gate, cache packages, drop the redundant main build - #531

Merged
tylerkron merged 1 commit into
mainfrom
claude/ci-performance-quick-wins-dd9a79
Nov 6, 2026
Merged

tylerkron merged 1 commit into
mainfrom
claude/ci-performance-quick-wins-dd9a79

Conversation

@tylerkron

Copy link
Copy Markdown
Contributor

Why

A pull request took ~7m30s wall clock and the iOS head job was ~7m04s of it. Every other job finished inside 3m40s, so iOS alone decided how long CI took — and inside it, one phase decided the job: 312 of the build step's 354 seconds passed between Optimizing assemblies for size and the next line of output.

job before
iOS head 7m 04s
Desktop head on macOS 3m 40s
Android head 3m 10s
Library + Desktop + capture 1m 57s
CI script self-tests / lockstep 10s / 8s

Runner queue waits were 2–3s, so contention was never the problem.

The main change: no LLVM on the simulator gate

The iOS SDK enables LLVM for every Release mobile build, and the comment sitting directly above the property that does so claims the value "is ignored when using the simulator". That is wrong for this build — a default simulator build writes aot-instances.dll.s.bc (61 MB), .s.opt.bc (58 MB) and .llvm.o (30 MB), and pushing that bitcode through LLVM is where the five minutes went.

Measured on clean builds on one machine (rm -rf bin obj on all three projects, RID-less restore, then the exact command the workflow runs):

wall clock
default (LLVM on) 162.7 s
-p:MtouchUseLlvm=false 43.1 s

AOT itself still runs, which is the coverage this job exists for (#105) — the Mono backend still emits aot-instances.dll.o and System.Private.CoreLib.dll.o; only the LLVM optimisation pipeline is skipped. Both builds produce 148 warnings / 0 errors, a 136 MB .app, a 254 KB Assets.car and 6 CFBundleIcon* keys, so the asset-catalogue assertion still proves what it proved before. Lock files untouched.

What is given up: LLVM's optimisation of native code this job compiles and throws away — a 32.7 MB app binary instead of 35.5 MB. Stated plainly in the workflow comment, because it means CI's configuration now differs from the shipping one on a second axis (codegen backend, on top of the trimming difference the csproj already records). If an iOS bug ever turns out to be LLVM-specific, that comment is where to look. The flag is on the command line rather than in the csproj so local and device builds keep LLVM.

Two smaller things, neither on the critical path

NuGet packages were never cached. All four jobs logged cache: false and re-downloaded the whole graph every run — and the macos and ios jobs did it twice, because their from-scratch-derivation guard restores again with --force-evaluate. The eight committed packages.lock.json files make the cache key exact rather than approximate, since locked mode already pins every resolved version to those same files.

The push trigger on main rebuilt a tree this workflow had already built. The ruleset on main carries merge_queue with max_entries_to_merge: 1 and SQUASH, so the merge_group run builds main-plus-that-one-PR — the commit that lands. Over one week: 58 pull_request, 13 merge_group, 13 push — perfectly paired, each push run re-spending ~10 minutes of macOS at this repository's 10× private-repo billing for no new information.

What makes dropping it safe is specifically that bypass_actors is empty, so nothing can reach main outside the queue. That is noted in both the workflow and the README, because if a bypass actor is ever added the trigger has to come back. The cost: commits on main no longer carry their own check status — the green tick lives on the queue entry that produced them.

Checked and rejected

  • AOT parallelism. AotCompileMaxDegreeOfParallelism is never set by the SDK, and the task does run strictly serially — one mono-aot-cross across 110 consecutive 1s samples on a 12-core box. But there are only 2 AOT units and aot-instances dominates utterly (61 MB of bitcode vs 19 KB for CoreLib), so parallelism buys nothing.
  • Debug instead of Release for iOS. Skips AOT entirely — that is the coverage, not the cost.
  • Caching the iOS/Android workload installs (27s / 21s). Means caching a mutated ~/.dotnet, brittle against the SDK pin, and not worth it once the LLVM change lands.
  • Cutting the macOS determinism harness from 5 reps to 3 (~68s of 10×-billed). Not on the critical path, and ci: gate the macOS capture harness on --determinism, the half we can prove today #191 chose 5 deliberately. Left alone.
  • Moving iOS to merge_group-only. Would halve PR feedback, but an iOS break would arrive as an ejected queue entry rather than a red PR. Left as a deliberate non-change.

Expected effect

iOS ~7m → ~2.5–3m, and the whole workflow ~7m30s → ~4m, with Desktop head on macOS becoming the critical path. Roughly a third of this workflow's spend goes away with the push trigger.

Verification

  • actionlint clean on build.yml.
  • All six .github/scripts self-tests pass.
  • Parsed the workflow and confirmed triggers are now merge_group / pull_request / workflow_dispatch, cache: true on all four setup-dotnet steps, and the iOS build carrying -p:MtouchUseLlvm=false.
  • Full clean iOS build with the workflow's exact command: 43.1s, 148 warnings, 0 errors, Assets.car 254168 bytes, 6 CFBundleIcon keys, no lock-file drift.
  • The real check is this PR's own run — it exercises all three changes, though the package cache is cold on the first run and will only show its saving on the second.

Also refreshed .github/merge-queue/README.md, which still said the queue was "blocked on plan" — it has been live since September. Replaced the enabling instructions with the settings actually in force and recorded which two of them the push-trigger removal depends on.

🤖 Generated with Claude Code

…undant main build

A pull request took ~7m30s wall clock and the iOS head was ~7m04s of it; every
other job finished inside 3m40s. So the iOS job was the only thing that decided
how long CI took, and inside it a single phase decided the job: 312 of the build
step's 354 seconds passed between "Optimizing assemblies for size" and the next
line of output.

That phase is LLVM. The iOS SDK enables it for every Release mobile build, and
the comment sitting directly above the property that does so claims the value
"is ignored when using the simulator" — which is wrong for this build. A default
simulator build writes aot-instances.dll.s.bc (61 MB), .s.opt.bc (58 MB) and
.llvm.o (30 MB), and pushing that bitcode through LLVM is where the five minutes
went.

Passing -p:MtouchUseLlvm=false in the workflow, measured on clean builds on one
machine (rm -rf bin obj on all three projects, RID-less restore, then the exact
command the workflow runs):

    default (LLVM on)        162.7 s
    -p:MtouchUseLlvm=false    43.1 s

AOT itself still runs, which is the coverage this job exists for (#105): the Mono
backend still emits aot-instances.dll.o and System.Private.CoreLib.dll.o, only
the LLVM optimisation pipeline is skipped. Both builds produce 148 warnings and
0 errors, a 136 MB .app, a 254 KB Assets.car and 6 CFBundleIcon* keys, so the
asset-catalogue assertion still proves what it proved before, and the lock files
are untouched. What is given up is LLVM's optimisation of native code this job
compiles and throws away — a 32.7 MB app binary instead of 35.5 MB. The flag is
on the command line rather than in the csproj so local and device builds keep
LLVM.

Two smaller things, neither on the critical path:

Nothing in this repo cached NuGet packages — all four jobs logged cache: false
and re-downloaded the whole graph every run, and the macos and ios jobs did it
twice because their from-scratch-derivation guard restores again with
--force-evaluate. The eight committed packages.lock.json files make the cache key
exact rather than approximate, since locked mode already pins every resolved
version to those same files.

The push trigger on main rebuilt a tree this workflow had already built. The
ruleset on main carries merge_queue with max_entries_to_merge 1 and SQUASH, so
the merge_group run builds main-plus-that-one-PR, which is the commit that lands;
over one week that was 13 push runs against 13 merge_group runs, perfectly
paired, each re-spending ~10 minutes of macOS at this repository's 10x billing.
What makes dropping it safe is specifically that bypass_actors is empty, so
nothing can reach main outside the queue — noted in both places, because if a
bypass actor is ever added the trigger has to come back. The cost is that commits
on main no longer carry their own check status.

The merge-queue README still said the queue was blocked on plan; it has been live
since September. Replaced the enabling instructions with the settings actually in
force, and recorded which two of them the push-trigger removal depends on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tylerkron
tylerkron added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit a8328db Nov 6, 2026
7 checks passed
@tylerkron
tylerkron deleted the claude/ci-performance-quick-wins-dd9a79 branch November 6, 2026 04:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant