Skip to content

Heap corruption when 2+ threads allocate finalizable objects (incl. WeakReference): memory access out of bounds in sgen_finalize_in_range #137903

Open

Description

@aopheim

Description

With WasmEnableThreads=true (Mono, browser-wasm; reproduced on both .NET 10.0.12 and .NET 11.0.0), allocating objects that need finalization on two threads at the same time corrupts the managed heap within a fraction of a second. "Need finalization" covers an ordinary class with a ~Finalizer(), System.WeakReference and WeakReference<T>. Usually a pthread worker then traps inside the GC with RuntimeError: memory access out of bounds, and the runtime goes down.

The same work on one thread almost always completes (see the table below), as does the same allocation load without finalizable objects. The repro is about 40 lines of plain C#, with no native libraries and no unsafe code.

How we found it: SkiaSharp keeps a System.WeakReference for every native object it wraps. As a result, any parallel rendering with SkiaSharp in a multithreaded Blazor WebAssembly app crashes this way, independent of what is being drawn.

Reproduction Steps

Prerequisites: .NET 10 SDK, the wasm-tools workload (dotnet workload install wasm-tools) and Node.js (only for the static file server).

  1. Create a project with the files below (also attached in full as WasmMtFinalizerRepro/).

    WasmMtFinalizerRepro.csproj

    <Project Sdk="Microsoft.NET.Sdk.WebAssembly">
      <PropertyGroup>
        <TargetFramework>net10.0</TargetFramework>
        <OutputType>Exe</OutputType>
        <Nullable>enable</Nullable>
        <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
        <WasmEnableThreads>true</WasmEnableThreads>
      </PropertyGroup>
    </Project>

    Program.cs (trimmed to the essential part here; the attached version adds the control modes listed under "Actual behavior")

    using System;
    using System.Collections.Generic;
    using System.Runtime.InteropServices.JavaScript;
    using System.Threading;
    using System.Threading.Tasks;
    
    Console.WriteLine("Repro loaded.");
    
    public static partial class Repro
    {
        [JSExport]
        internal static Task<string> Run(string mode, int threads, int iterations)
            => Task.Run(() =>
            {
                long checksum = 0;
                Parallel.For(0, threads, new ParallelOptions { MaxDegreeOfParallelism = threads }, t =>
                {
                    var keep = new List<object>(256);
                    var rnd = new Random(t);
                    long local = 0;
                    for (int i = 0; i < iterations; i++)
                    {
                        var arr = new float[rnd.Next(4, 512)];
                        local += arr.Length;
    
                        if (mode == "finalizable") { var f = new Finalizable(arr); if (f.Target is not null) local++; }
                        if (mode == "weakref")     { var w = new WeakReference(arr); if (w.IsAlive) local++; }
    
                        keep.Add(arr);                       // keep a few arrays alive across nursery GCs
                        if (keep.Count == 256) keep.Clear();
                    }
                    Interlocked.Add(ref checksum, local);
                });
                return $"done gen0={GC.CollectionCount(0)} checksum={checksum}";
            });
    }
    
    public sealed class Finalizable(object target)
    {
        public object Target = target;
        ~Finalizable() { }
    }

    wwwroot/main.js (trimmed; the attached one adds timing and a crash listener)

    import { dotnet } from './_framework/dotnet.js';
    const q = new URLSearchParams(location.search);
    const { getAssemblyExports, getConfig, runMain } = await dotnet.create();
    await runMain();
    const exports = await getAssemblyExports(getConfig().mainAssemblyName);
    console.log(await exports.Repro.Run(q.get('mode') ?? 'finalizable', Number(q.get('threads') ?? 2), 2_000_000));

    wwwroot/index.html: a page that loads main.js as a module.

  2. Publish: dotnet publish -c Release -o publish

  3. Serve publish/wwwroot with COOP same-origin and COEP require-corp. The attached serve.mjs does that: node serve.mjs publish/wwwroot 5081

  4. Open these URLs in Chrome:

    • http://localhost:5081/?mode=finalizable&threads=2: crashes.
    • http://localhost:5081/?mode=weakref&threads=2: crashes.
    • http://localhost:5081/?mode=none&threads=2: completes.
    • http://localhost:5081/?mode=weakref&threads=1&iterations=4000000: completes, with the same total work on one thread. The iterations parameter exists only in the attached main.js.

The attached harness.html runs any configuration N times, each time in a fresh iframe, and tallies the results: harness.html?mode=weakref&threads=2&trials=10.

Two more URL parameters in the attached main.js:

  • runner=parallel|tasks|threads picks how the workers start: Parallel.For (the default), one Task.Run each plus await Task.WhenAll, or one dedicated new Thread each. On .NET 10 the crash happens with both parallel and tasks, so it does not depend on Parallel.For.
  • pool=16 calls dotnet.withConfig({ pthreadPoolInitialSize: 16 }). This is needed on .NET 11. Without it, no thread created after startup ever runs (#136492), so every 2-thread run hangs, even the control. With it, .NET 11 crashes the same way as .NET 10: http://localhost:5081/?mode=finalizable&threads=2&pool=16.

To build for .NET 11, change TargetFramework to net11.0. Use hook-worker-net11.sh for stacks: Emscripten 6 no longer ships a separate worker script.

Expected behavior

All modes run to completion and print a result, as the single-threaded runs do.

Actual behavior

Mostly, within 0.1–0.6 s of starting the run, a worker traps and the whole runtime goes down:

worker sent an error! .../dotnet.native.worker.3pea7uf3ce.mjs:88: Uncaught RuntimeError: memory access out of bounds
Uncaught (in promise) RuntimeError: memory access out of bounds

By default the worker's stack is not forwarded. With a symbolized build and a small diagnostic patch to the worker script (hook-worker.sh, attached), the stack is:

RuntimeError: memory access out of bounds
    at copy_object_no_checks.1
    at simple_nursery_serial_copy_object
    at sgen_finalize_in_range
    at finish_gray_stack
    at collect_nursery
    at sgen_perform_collection
    at sgen_ensure_free_space
    at sgen_alloc_obj_nolock
    at mono_gc_alloc_vector
    at mono_array_new_specific_internal
    at mono_array_new_specific_checked
    at mono_interp_exec_method
    ...
    at start_wrapper

Other symptoms of the same corruption, seen with the same repro:

  • * Assertion: should not be reached at /__w/1/s/src/runtime/src/mono/mono/sgen/sgen-scan-object.h:93 in major_scan_object_with_evacuation ← drain_gray_stack ← finish_gray_stack ← major_do_collection. This was followed by "MONO_WASM: forcing abort 3000ms after last error log message", after which the page's main thread stopped responding.
  • On AOT, FATAL UNHANDLED EXCEPTION: System.ArrayTypeMismatchException at System.Threading.ThreadInt64PersistentCounter.CreateThreadLocalCountObject() on a thread-pool thread. That thread never acknowledged the next suspend request (state 0x105), so the stop-the-world gave up with WAITING for 1 threads, got 0 suspended and an assertion at mono/utils/mono-threads.c:336. The same suspend-timeout assertion also happened once on the interpreter.
  • In the original application (SkiaSharp), two more stacks appeared. One was sgen_client_object_finalize_eagerly ← sgen_finalize_in_range ← finish_gray_stack ← collect_nursery, the trap inside the eager finalization of a WeakReference. The other was copy_object_no_checks ← simple_nursery_serial_scan_object ← simple_nursery_serial_drain_gray_stack ← finish_gray_stack.

Full logs, including the thread dump, are attached in logs/.

Results. Each row is a mode string, and a comma-separated mode gives each thread its own mode. "Load" means one page load. Each run is iterations allocations per thread: 2,000,000 with 2 threads, 4,000,000 with 1 thread. Every completed run performed about 1,020–1,050 gen0 collections. Unless stated otherwise, the runner is parallel.

.NET 10.0.12

Build Mode Threads Result
Interpreter weakref 2 20 of 20 loads failed (19 OOB trap, 1 suspend-timeout assertion)
Interpreter weakref-generic (WeakReference<float[]>) 2 6 of 6 loads failed (OOB)
Interpreter finalizable 2 13 of 13 loads failed (12 OOB, 1 sgen-scan-object.h:93 assertion)
Interpreter finalizable,none (one thread finalizable, one only arrays) 2 2 of 2 loads failed (OOB)
Interpreter weakref,none (one thread WeakReference, one only arrays) 2 6 of 6 runs completed
Interpreter gchandle-weak (GCHandle.Alloc(arr, Weak) + Free()) 2 5 of 5 runs completed
Interpreter gchandle-normal 2 6 of 6 runs completed
Interpreter none 2 14 of 14 runs completed
Interpreter weakref 1 13 runs completed; 1 load failed (OOB, about 3 s in)
Interpreter finalizable 1 1 run completed
AOT weakref 2 6 of 7 loads did not complete: 4 OOB, 1 ArrayTypeMismatchException plus suspend timeout, 1 not finished after 6 s (not examined); 1 completed
AOT finalizable 2 2 of 2 loads failed (OOB)
AOT none 2 3 of 3 runs completed
AOT weakref 1 3 of 3 runs completed
Interpreter, runner=tasks finalizable 2 1 of 1 load failed (OOB)
Interpreter, runner=tasks none 2 2 of 2 runs completed (workers overlapped on two threads)

.NET 11.0.0 (SDK 11.0.100, Emscripten 6.0.2). All 2-thread rows use pool=16.

Build Mode Threads Result
Interpreter weakref 2 6 of 8 loads failed (OOB). In the other 2, the runtime had not finished starting after 5 s (not examined).
Interpreter finalizable 2 4 of 4 loads failed (OOB)
Interpreter weakref,none 2 1 of 1 load failed (OOB, 7.5 s after page load)
Interpreter gchandle-weak 2 3 of 3 runs completed
Interpreter none 2 2 of 2 runs completed (workers overlapped on two threads)
Interpreter weakref 1 10 of 10 runs completed (4 of them in the symbolized build)
AOT weakref 2 1 of 3 loads failed (OOB). The other 2 gave no result within 8–10 s (not examined).
AOT finalizable 2 2 of 2 loads failed (OOB)
AOT none 2 3 of 3 runs completed
AOT weakref 1 6 runs completed; 1 load failed (OOB, at the start of its 2nd run)
Any anything 2 Without pool: 0 completed. Every 2-thread run hangs, even mode=none&iterations=0. See #136492.

What the tables show:

  • The trigger is allocating finalizable objects (WeakReference and WeakReference<T> have finalizers) while another thread also allocates. Raw weak GC handles alone don't trigger it.
  • One finalizable-allocating thread next to a plain-allocating thread is enough. finalizable,none crashed on .NET 10, and weakref,none crashed on .NET 11 (it survived 6 runs on .NET 10). Few runs were done for these rows.
  • A single thread is not entirely safe. On both versions about 1 in 15 single-thread weakref runs also failed.
  • .NET 11 behaves the same once [browser][MT] getNewWorker never finds a loaded worker after the Emscripten 5.0.6 upgrade (worker.loaded no longer set); first Task.Delay/timer deadlocks JS→.NET dispatch on 11.0 rc.3 #136492 is worked around. On .NET 11 the symbolized finalizable crash has exactly the .NET 10 stack (copy_object_no_checks ← sgen_finalize_in_range ← finish_gray_stack ← collect_nursery). Two weakref crashes instead trapped in a mutator: m_class_alloc0 ← get_virtual_method_fast ← mono_interp_exec_method, a virtual call on an object whose class pointer is garbage. See logs/5-net11-stacks.txt.

Regression?

Not a recent regression: it happens on both .NET 10.0.12 and .NET 11.0.0. We have not tried older versions. Multithreaded WebAssembly is still marked experimental. Single-threaded builds of the same app are fine.

Known Workarounds

None in application code, other than not allocating finalizable objects or WeakReferences on more than one thread. With SkiaSharp that rules out parallel rendering entirely. Running all such work on a single background thread works, but with no parallelism.

Configuration

  • .NET SDK 10.0.102, wasm-tools workload 10.0.112. Runtime pack Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm 10.0.12 (dotnet/dotnet@95017c7). Emscripten 3.1.56 (Microsoft.NET.Runtime.Emscripten.3.1.56.* 10.0.12).
  • dotnet publish -c Release. Fails both with the interpreter (default) and with -p:RunAOTCompilation=true. Native relink happens because the workload is installed.
  • Windows 11 Pro x64 (10.0.26200), 8 logical cores.
  • Browser: Chromium 156.0.8078.75 (the Chromium embedded in an Electron desktop app). We did not try standalone Chrome, Firefox or Safari, nor Node.
  • .NET 11: SDK 11.0.100 (installed with dotnet-install.ps1 -Channel 11.0 into a separate folder), wasm-tools workload 11.0.100.1. Runtime pack Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm 11.0.0. Emscripten 6.0.2. Same OS and browser. Pages opened with pool=16 (see Reproduction Steps).

Other information

The rest of this section is analysis from reading release/10.0. It is not verified, so please treat it as a hypothesis.

How registration works

  • A class with a finalizer never gets the managed allocator (sgen-mono.c#L851-L852).
  • It is allocated via mono_object_new_alloc_specific_checked → object_new_common_tail (object.c#L5250-L5268). That passes the new object to mono_object_register_finalizer as a raw MonoObject*, which ends in sgen_object_register_for_finalization → add_stage_entry (sgen-fin-weak-hash.c#L430-L567).
  • WeakReference and WeakReference<T> have finalizers, so they take this path too. WeakReference is additionally finalized eagerly inside the GC (SGEN_GC_BIT_WEAKREF, sgen_client_object_finalize_eagerly).

Where it could go wrong

  • If another thread is draining the stage queue (next_fin_stage_entry == -1, set by try_lock_stage_for_processing when the queue overflows), add_stage_entry spins in mono_thread_info_usleep(200). That function does MONO_ENTER_GC_SAFE. Contention on LOCK_GC in the overflow path can also enter GC-safe mode.
  • So a registering thread can be GC-safe while it holds the only reference to the new object, in a C local. At that point the object is neither in the stage queue nor in the finalizable hash.
  • On WASM, C/wasm locals are not scanned conservatively; compare the closed-as-by-design #131466. If a nursery collection runs then, the object can be collected or moved. The thread then stores a stale pointer into the stage entry.
  • collect_nursery moves stage entries into the finalizable hash (sgen-gc.c#L1851), and sgen_finalize_in_range → copy_func or sgen_client_object_finalize_eagerly would then work on garbage. That matches the stacks above.

What this explains, and what it doesn't

  • It would explain why concurrency matters, since some thread has to hold the GC lock or drain the queue at the right moment.
  • It would also explain why GCHandle loops are unaffected: they never touch the stage queue.
  • It does not obviously explain the cases with only one registering thread: finalizable,none, weakref,none on .NET 11, and the rare single-thread failure. Other runtime threads (finalizer, thread pool) may register objects too, or the window may be reachable without a second filler.

Possibly related, but none of these match: #104524, #97543, #110805, #65242. No existing issue mentions sgen_finalize_in_range or finalizable allocation on browser MT.

WasmMtFinalizerRepro.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions