Test failure: GC/Scenarios/THDChaos/thdchaos/thdchaos.cmd

[VincentBu]

Failed in: runtime-coreclr pgostress 20260703.1

Failed tests:

coreclr linux arm Checked fullpgo_random_gdv @ (Debian.13.Arm32.Open)AzureLinux.3.Arm64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-13-helix-arm32v7@sha256:7cd2e25705d98e003a2d4f2e10cbf27847f573d9548e06a72062028ed74f43a6
- GC/Scenarios/THDChaos/thdchaos/thdchaos.cmd

Error message:

Unhandled exception. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, BOOL isThreadPool, Char* pThreadName, ObjectHandleOnStack exception)
   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, BOOL isThreadPool, Char* pThreadName, ObjectHandleOnStack exception)
   at System.Threading.Thread.StartCore()
   at System.Threading.Thread.Start(Boolean captureContext)
   at System.Threading.Thread.Start()
   at DefaultNamespace.LivingObject.ThreadStart()
   at System.Threading.Thread.StartCallback(Thread* pThread)Unhandled exception. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, BOOL isThreadPool, Char* pThreadName, ObjectHandleOnStack exception)
   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, BOOL isThreadPool, Char* pThreadName, ObjectHandleOnStack exception)
   at System.Threading.Thread.StartCore()
   at System.Threading.Thread.Start(Boolean captureContext)
   at System.Threading.Thread.Start()
   at DefaultNamespace.LivingObject.ThreadStart()
   at System.Threading.Thread.StartCallback(Thread* pThread)
Unhandled exception. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, BOOL isThreadPool, Char* pThreadName, ObjectHandleOnStack exception)
   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, BOOL isThreadPool, Char* pThreadName, ObjectHandleOnStack exception)
   at System.Threading.Thread.StartCore()
   at System.Threading.Thread.Start(Boolean captureContext)
   at System.Threading.Thread.Start()
   at DefaultNamespace.LivingObject.ThreadStart()
   at System.Threading.Thread.StartCallback(Thread* pThread)
Unhandled exception. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, BOOL isThreadPool, Char* pThreadName, ObjectHandleOnStack exception)
   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, BOOL isThreadPool, Char* pThreadName, ObjectHandleOnStack exception)
   at System.Threading.Thread.StartCore()
   at System.Threading.Thread.Start(Boolean captureContext)
   at System.Threading.Thread.Start()
   at DefaultNamespace.LivingObject.ThreadStart()
   at System.Threading.Thread.StartCallback(Thread* pThread)
Unhandled exception. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, BOOL isThreadPool, Char* pThreadName, ObjectHandleOnStack exception)
   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, BOOL isThreadPool, Char* pThreadName, ObjectHandleOnStack exception)
   at System.Threading.Thread.StartCore()
   at System.Threading.Thread.Start(Boolean captureContext)
   at System.Threading.Thread.Start()
   at DefaultNamespace.LivingObject.ThreadStart()
   at System.Threading.Thread.StartCallback(Thread* pThread)

Unhandled exception. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, BOOL isThreadPool, Char* pThreadName, ObjectHandleOnStack exception)
   at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, BOOL isThreadPool, Char* pThreadName, ObjectHandleOnStack exception)
   at System.Threading.Thread.StartCore()
   at System.Threading.Thread.Start(Boolean captureContext)
   at System.Threading.Thread.Start()
 

Stack trace:

   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor49|0_50(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

Kusto query result:

Queued	Pipeline	Pipeline Configuration	OS	Arch	Test	Outcome	Properties	Build Reason
2026-05-29 18:42:27.481000+00:00	runtime-interpreter	Checked-interpmode3_no_tiered_compilation	osx.15.amd64.open	x64	GC/Scenarios/THDChaos/thdchaos/thdchaos.cmd	Failed	{"operatingSystem":"OSX.15.Amd64.Open","architecture":"x64","System.StageAttempt":"1","System.JobName":"__default","System.JobAttempt":"1","BuildId":"1440788","AzurePipelinesTestRunId":"39954266","CollectionUri":"https://dev.azure.com/dnceng-public/","DefinitionName":"runtime-interpreter","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.PhaseName":"run_test_p1__osx_x64_checked","System.PhaseAttempt":"1","configuration":"Checked-interpmode3_no_tiered_compilation","BuildNumber":"20260529.1","Reason":"Schedule","System.StageName":"Build","DefinitionId":"316","Project":"public"}	Schedule

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @VSadov
See info in area-owners.md if you want to be subscribed.

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• Test failure: GC/Scenarios/THDChaos/thdchaos/thdchaos.cmd #119700

[JulieLeeMSFT]

cc @AndyAyersMS.

[VSadov]

@eduardo-vp - it could be another case where we create too many threads too fast and arm32 runs out of memory. We have disabled for arm32 some tests that are too eager with thread creation, or have put a lower limit on thread count created.

But it seems to be specific to

runtime-interpreter | Checked-interpmode3_no_tiered_compilation -- | --

So there is a chance it is a real issue, and that needs to be confirmed.

[janvorli]

@VSadov the arm32 failure has nothing to do with interpreter. The Kusto query result above is from osx x64.

[VSadov]

FWIW:

Why Thread.Start() throws OutOfMemoryException
For Thread.Start(), OutOfMemoryException can mean failure to allocate native thread resources, including:

native stack reservation/commit;
32-bit virtual address space;
OS thread metadata;
container memory;
an OS process/thread limit.
This is particularly plausible on Arm32, where address space is constrained. The fact that many concurrent calls fail specifically in Thread.StartInternal strongly supports resource exhaustion.

There is also a synchronization bug
iCounter is static and shared by every LivingObject, but the increment uses lock(this):

dotnet / runtime / src / tests / GC / Scenarios / THDChaos / livingobject.cs
v3
public void IncreatCount()
{
    lock(this)
    {
        iCounter += 1;
    }
Each LivingObject has a different this, so increments from different objects are not mutually exclusive. Consequently:

increments can be lost;
the threshold check is unsynchronized;
multiple threads can concurrently observe iCounter < 20;
more successor threads can be created than intended.
This probably amplifies the burst, although the unconditional 500 thread starts are already enough to make the test fragile.

Conclusion
Confidence: high that this failure is caused by the test rapidly creating hundreds of native threads and exhausting Arm32/container resources. It does not, by itself, look like a GC heap OOM or necessarily a runtime regression.

A sensible mitigation would be to:

reduce the iteration/thread count on Arm32;
use Interlocked.Increment(ref iCounter) for the global counter;
reserve a thread-creation slot atomically before calling Start;
optionally limit concurrency or join completed threads rather than launching roughly 500 at once.