perf: establish staged upstream-parity and profiler optimization program

[zackees]

Context

The fork is entering a performance phase with one non-negotiable constraint: diagnostic and profiling work must not make normal release allocation slower. We need to compare like-for-like builds, find unintended serialization, profile before optimizing, and preserve correctness, portability, memory use, and tail latency while pursuing speedups.

There is already evidence that the compiled-in-but-stopped profiler is not free. The Release microbenchmark recorded in README/PR #154 measured 11.75 ns/allocation with MI_PPROF=OFF versus 20.00 ns/allocation with MI_PPROF=ON and profiling stopped (about +70%). The current allocation fast path calls _mi_prof_on_alloc, which performs an atomic enabled check; active samples later serialize record creation, stack interning, and page metadata updates under the global prof_lock. The free path usually checks page->has_metadata before taking that lock.

Several confounders must be isolated rather than attributed to pprof:

• The fork's independent memory-events feature is always compiled and adds one relaxed state check to allocate/free/realloc even when runtime-disabled. Therefore fork MI_PPROF=OFF is not structurally identical to upstream.
• MI_PPROF=ON adds -fno-omit-frame-pointer on non-Windows, which can change code generation independently of the hook.
• mi_page_t contains profiler metadata fields and mi_tld_t contains profiler sampling state even in an MI_PPROF=OFF build. Their size, alignment, and cache-line placement must be compared with upstream.
• Upstream already has optional MI_OPT_ARCH and MI_OPT_SIMD paths, including AVX2 bitmap scans. We must measure those before inventing overlapping SIMD implementations.
• The live upstream/dev3 ref can move. The parity baseline for this program is the fork overlay's exact pinned base, bcee5a88, unless a separate issue-scoped overlay update is reviewed and verified first.

GitHub-hosted runners are suitable for correctness and smoke tests, but not for accepting small timing wins. Performance claims require repeatable measurements on stable machines.

Proposal

Run this as ordered phases. Do not start a later optimization phase until the preceding phase has produced reproducible evidence and its regressions are fixed. Use one PR per phase; keep Rust harness and C-core changes in separate commits as required by the repository.

Phase 0 — trustworthy Rust benchmark and profiling harness

Extend/reuse the native-thread Rust harness from #168 rather than using Python to generate allocator load. Python may remain as a build/report orchestrator, but timed operations and concurrency must execute in native Rust/C code.

The harness should:

• build and run separate Release artifacts to avoid linking multiple mimalloc implementations into one process;
• record source commit, compiler/version, flags, MI_PPROF, frame-pointer setting, MI_OPT_ARCH, MI_OPT_SIMD, runtime options, CPU/OS, affinity, frequency/power policy, seed, and workload parameters;
• use native threads, deterministic seeds, barriers, fixed worker affinity where supported, warmups, randomized/interleaved A/B order, multiple samples, confidence intervals, and outlier/noise reporting;
• emit machine-readable results plus a short human summary and exact reproduction command;
• measure throughput/latency (including p95/p99), RSS/commit/peak memory, fragmentation where observable, and hardware counters where available: cycles, instructions, IPC, branches/misses, cache misses, and stalled cycles;
• include single-thread and scaling runs, same-thread and cross-thread free, fixed and mixed sizes, hot-cache and cold/working-set pressure, aligned/realloc/zeroing, burst/sawtooth, thread churn, and representative real-world traces or mimalloc-bench workloads;
• support a planted slower/serialized control that the statistical comparison must reject;
• add no release C dependency or runtime diagnostic work.

Use stable Windows x64 (MSVC and MinGW builds) and Linux x64 reference hosts for performance evidence. Keep macOS and other supported targets in correctness/smoke coverage. Record the reference machines in the first baseline rather than silently comparing results across different hardware.

Phase 1 — fork Release parity with pinned upstream dev3

Compare pinned upstream dev3@bcee5a88 with fork Release MI_PPROF=OFF, with memory-events runtime-disabled. Produce size/alignment/cache-line and disassembly diffs for hot allocator structures and functions as well as runtime measurements.

Attribute every statistically meaningful difference. In particular, separate:

• always-on memory-events hook cost;
• unguarded profiler-related structure layout changes;
• code layout/inlining/instruction-cache effects caused by fork edits;
• compiler, LTO, frame-pointer, and build-system differences;
• behavioral differences in allocator options or initialization.

Fix confirmed regressions before moving on. A fork feature may retain an explicitly measured cost only if it is unavoidable, documented, and accepted in the issue; the default goal is parity with no unexplained slowdown.

Phase 2 — pprof compiled in, runtime stopped

Compare identical fork Release builds with MI_PPROF=OFF and MI_PPROF=ON while profiling is stopped. On non-Windows, add control builds that independently toggle frame-pointer omission so the hook and unwind-ready code-generation costs are not conflated.

Look adversarially for surprising serialization and shared-cache traffic, not just locks: atomic flag loads, cache-line bouncing, TLS access, lost inlining, register pressure, branch-predictor changes, hot function growth, and instruction-cache displacement. Confirm findings with disassembly and counters.

Evaluate the already-documented Bun-style alternative: keep the normal allocation fast path byte-identical while profiling is stopped, and switch/direct pages to a profiling-aware cold/generic route only when profiling starts. Any design must preserve correct activation for existing threads and heaps and restore the fast path safely on stop.

Phase 3 — pprof active

Profile enabled pprof across sample intervals, stack depths, thread counts, allocation sizes, cross-thread frees, dump/snapshot activity, and short/long-lived allocations. Publish scaling curves and flamegraphs/counter data that separately attribute:

• the thread-local sampling decision;
• stack capture/unwinding;
• the global record/free lists;
• stack hashing/interning and table growth;
• page metadata attach/free lookup;
• global atomics and prof_lock wait/hold time;
• dump/snapshot encoding and symbol/map work;
• raw-OS profiler arena allocation/commit behavior.

Optimize only measured bottlenecks. Candidate experiments include per-thread sample buffers, batched publication or range reservation, sharded stack/record tables, thread-local or small local stack-dedup caches, compact PC storage, deferred symbolization/encoding, bounded snapshot handoff, and moving cold dump/configuration code away from allocator hot text. All profiler-internal memory must continue to come only from the raw OS layer.

Phase 4 — portable low-risk allocator wins

After parity and profiler costs are understood, pursue low-hanging release improvements one at a time. Candidate areas:

• remove redundant loads, atomics, branches, and option checks;
• improve hot/cold function and field placement without enlarging critical cache-line footprints;
• reduce false sharing and cross-core ownership traffic;
• batch per-thread accounting/publication;
• improve free-list, size-class, page-map, and bitmap lookup locality;
• reduce code size or restore profitable inlining where measurements show front-end pressure;
• specialize common fixed-size/batch allocation paths while retaining a portable fallback.

Each candidate needs its own before/after evidence and must be reverted if the effect is noise or merely shifts cost to another representative workload.

Phase 5 — cache warming and prefetch experiments

Prefetching is speculative and can easily make an allocator slower through wasted bandwidth, cache eviction, instruction pressure, false sharing, NUMA traffic, or premature/speculative page faults. Treat each site as an isolated experiment with a no-prefetch control and distance sweep.

Plausible sites to measure include:

• the next free-list node during batch allocation/free;
• page metadata/page-map entries after the owning page becomes known;
• the next bitmap word or arena chunk during multiword scans;
• remote-free list heads before collection;
• destination cache lines immediately before large zero/fill operations;
• the next page only when a stable sequential allocation pattern predicts imminent use.

Test read prefetch, write-intent prefetch where supported, and compiler-generated behavior. Reject a prefetch if it does not win across the intended working-set/latency cases or if it worsens bandwidth, remote NUMA traffic, tails, or unrelated workloads. Do not add unconditional prefetches to the common path based only on a microbenchmark.

Phase 6 — architecture-specific deeper wins

First benchmark the existing baseline, MI_OPT_ARCH, and MI_OPT_ARCH + MI_OPT_SIMD configurations. Audit generated code for already-available TZCNT/LZCNT, POPCNT, BMI1/BMI2, conditional-move, AVX2 bitmap, and ARM64 LSE/NEON opportunities before adding new paths.

Only then consider targeted implementations such as vectorized bitmap/record scans, batched hash comparisons, wider zero/copy paths, non-temporal stores for sufficiently large cold destinations, or architecture-specific prefetch/write-intent hints. Requirements:

• portable scalar fallback with identical semantics;
• no feature check on the allocation hot path (select once at build/init or use a reviewed dispatch scheme);
• MSVC and MinGW support for x86 changes, plus existing supported-platform correctness;
• dedicated evidence for code-size/front-end cost and AVX frequency/downclock effects;
• no raised ISA floor for default release artifacts unless explicitly approved;
• no regression when the specialized path is unavailable or loses on small inputs.

Acceptance criteria

• A native-thread Rust performance harness (building on test: prefer a Rust harness for concurrent stress and soak testing #168) produces deterministic, machine-readable, reproducible A/B results and rejects a planted serialized/slower control.
• The comparison matrix includes pinned upstream, fork MI_PPROF=OFF, fork MI_PPROF=ON stopped, and representative active-profiler configurations; memory-events, frame pointers, MI_OPT_ARCH, and MI_OPT_SIMD are independently controlled.
• Phase 1 reports and explains hot structure sizes/alignment/cache-line placement, disassembly/code size, and runtime/counter differences. No unexplained upstream-parity regression remains.
• Phase 2 quantifies stopped-profiler overhead and removes or explicitly resolves unintended serialization/shared-cache traffic. The normal release path is not made slower to improve diagnostics.
• Phase 3 publishes active-profiler flamegraphs/scaling/counter evidence and attributes sampling, unwinding, interning, record/page bookkeeping, locking, dumping, and profiler-arena costs before changes are proposed.
• Every optimization has RED -> GREEN evidence: the pre-change benchmark reproduces a statistically significant regression/bottleneck, and the same paired protocol shows the improvement after the change.
• Report confidence/noise and practical effect size; do not accept a win from a single run. Thresholds are derived from the recorded reference host's noise floor and validated by the planted control.
• Every accepted change also passes representative non-target workloads with no material throughput, p99 latency, RSS/commit, fragmentation, or scaling regression.
• Prefetch/cache-warming changes include a no-prefetch control, distance sweep, and cache/bandwidth/NUMA evidence; inconclusive changes are not shipped.
• ISA-specific changes benchmark the existing architecture/SIMD modes first, retain the default portable fallback, avoid per-allocation dispatch, and pass MSVC and MinGW.
• All repository merge gates remain green (c-unit ON/OFF matrix and rust-native), profiler memory continues to use only the raw-OS arena, and no required C-build dependency is added.
• Each phase posts its exact commands, raw result artifact, summary, and hardware/build metadata to this issue before the next phase begins.

Open questions

• Which stable Windows and Linux machines should be designated as the primary reference hosts? Record the first chosen hosts and do not compare small deltas across unlike hardware.
• Should optional x86 optimizations remain compile-time MI_OPT_ARCH/MI_OPT_SIMD variants, or should a later phase evaluate one-time runtime dispatch for distributable binaries? Decide only after the compile-time variants show a robust win.
• What maximum stopped-profiler overhead is acceptable if a byte-identical zero-cost route proves infeasible? Establish this from the harness noise floor and production use cases rather than choosing an arbitrary percentage now.

Related issues

• test: prefer a Rust harness for concurrent stress and soak testing #168 — native Rust concurrent stress/soak harness; this performance harness should reuse its execution, replay, watchdog, and machine-readable infrastructure.
• META: mimalloc v3 fork landscape — strategies, upstream gaps, and what we should adopt #50 — completed fork survey documenting the +70% stopped-profiler result, Bun's zero-cost-when-off routing strategy, and existing SIMD/benchmark prior art.

[zackees]

Phase 0 complete — merged in #173.

rust/bench-harness crate added:

• Warmup + measurement round separation with per-round timing
• Statistical aggregation (mean, std-dev, CV) and Welch t-test A/B comparison
• Build + host metadata recording (commit, rustc, CPU, OS, timestamp)
• Planted-slower Mutex-serialized control: normal 3.4×–12.6× throughput vs serialized, reliably rejected at α=0.05 with 7 measurement rounds
• Machine-readable JSON output with reproduction command
• All 17 CI checks green on all platforms (MSVC, MinGW, Linux ARM/x86_64, macOS Intel/ARM)

Phase 1-6 remain as follow-up work.