Repository navigation
Description
Description
On NVIDIA Jetson AGX Xavier (8 Carmel ARMv8.2 cores, Linux 5.10 from NVIDIA's L4T), the whole CPU complex stops within seconds of a .NET process starting. Nothing is printed on any console, including the serial console, and the board's hardware watchdog resets it about two minutes later (reset source BCCPLEXWDT). We see it on every L4T release we run (R35.1.0, R35.4.1, R35.6.5, R35.7.0 = JetPack 5.0.2 to 5.1.8), on NVIDIA's developer kit and on a third-party carrier board, in CI and on devices in the field.
We think the root cause is most likely below the OS (CPU or its firmware), so this is not a claim of a runtime bug. We are reporting it because a .NET process is involved in every freeze we have recorded, we have not been able to trigger it without .NET, and the first occurrence in our CI history coincides with a runtime update. We hope someone recognises the pattern, or can say which runtime behaviour could be switched off to avoid it.
Reproduction Steps
On an AGX Xavier with all 8 cores online, in a loop:
sync; echo 3 > /proc/sys/vm/drop_caches(cold page cache)dotnet build tests/tests.csproj -c Release --no-restore --no-incremental -m:8 -nodeReuse:false -p:UseSharedCompilation=falseon a small solution (8 class libraries of 200 trivial methods each plus one xUnit project referencing them; solution created withdotnet newby the same SDK)- stop the build after 9 seconds (every freeze we recorded came within about 5 s of the build starting) and kill its processes
In a "bad" period this freezes the machine about once in 10 to 40 rounds. The rate changes over hours under the same load for reasons we do not understand, which makes A/B tests slow.
Expected behavior
The build runs or fails; the machine keeps running.
Actual behavior
The whole machine stops. A kernel module that samples each busy core's program counter every 100 µs into memory that survives the reset shows all busy cores stopping within about 0.1 ms of each other, each in ordinary user or kernel code (no common function, no cache maintenance in progress, sometimes GC running and sometimes not). Idle cores leave idle at the end. The kernel prints nothing: no soft lockup, no RCU stall, no panic.
Regression?
In our CI on two AGX Xavier runners, jobs began to be lost this way on 13 May 2026, the day the repositories moved from SDK 10.0.202 (runtime 10.0.6) to 10.0.300 (runtime 10.0.8) and started installing the SDK fresh for every job (so every job starts from a cold page cache). None were lost in the months before. That is a correlation, not proof: it also coincides with the cold-cache change.
A same-hours comparison on one machine (all cores, cold cache, SDKs in turn): SDK 10.0.202 0 freezes in 234 rounds, SDK 10.0.300 3 in 240, SDK 11.0.100 0 in 237. Too few freezes to be conclusive; a runtime-only swap (SDK 10.0.300 with runtime 10.0.6 against 10.0.8) is running now.
Known Workarounds
None that we trust. With CPUs 1 to 7 taken offline (CPU hotplug) it froze 0 times in 175 rounds where all cores froze 6 times in 103, but it has frozen once with one core online. Four cores online also froze.
Did not prevent it (each froze at least once with the setting):
DOTNET_EnableWriteXorExecute=0DOTNET_TieredCompilation=0DOTNET_gcConcurrent=0DOTNET_UseGCWriteBarrierCopy=0DOTNET_EnableArm64Atomics=0DOTNET_ReadyToRun=0- an
LD_PRELOADhook around every__clear_cachecall of libcoreclr - reading all SDK files into the page cache before the build
- pinning the whole build to one core while the others stay online
- kernel
mitigations=off,nospectre_v2 nokaslr kpti=off,transparent_hugepage=never - fixed maximum CPU/EMC/GPU clocks, idle states disabled, CPU firmware background optimisation disabled
One freeze came 0.1 s after execve of SDK 11's dotnet build, inside the NativeAOT host, while the GC was committing its first regions (mprotect and madvise pairs 4 MiB apart), before any JIT.
Without .NET it never froze: programs that write and run millions of new functions a second on all cores, 89 million new executable pages, 620,000 process starts, cold starts of system binaries, CPU hotplug without .NET.
Configuration
- NVIDIA Jetson AGX Xavier, L4T R35.1.0 to R35.7.0, Linux 5.10.x-tegra, Ubuntu 20.04 arm64
- .NET SDK 10.0.202, 10.0.300, 11.0.100 (runtime 10.0.6, 10.0.8, 11.0.0), linux-arm64
- Not specific to one carrier board
Other information
Questions we would be grateful for help with:
- Did anything change between runtime 10.0.6 and 10.0.8 on linux-arm64 in how the runtime issues process-wide memory barriers (
membarrier/FlushProcessWriteBuffers), cache maintenance, or GC region commit/decommit, that could be switched off with a setting? - Is a pattern like this known on other ARMv8 cores?
We can run any configuration or instrumentation you suggest.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsNo status
Activity
dotnet-policy-service commentedon Nov 19, 2026
Tagging subscribers to this area: @anicka-net, @dotnet/gc
See info in area-owners.md if you want to be subscribed.
janvorli commentedon Nov 19, 2026
@peters I am not aware of anything that would change in the process wide memory barriers or cache maintenance. And nothing in the GC either.
Can you try to run it under strace (with logging redirected to a file via the
-ocommand line option) to see if it would reveal anything?peters commentedon Nov 20, 2026
Thanks. Here is the strace result. This comment replaces my two earlier ones.
Setup.
strace -f -tt -s 48on the wholedotnet build. The output is piped off the board over the network, because nothing written locally survives the reset.Result. The hang still occurs under strace: 2 hangs in 21 traced builds. The first trace ends about 1 ms before the hang. Its last line is at 01:53:44.6915, and our kernel recorder's last event is at 01:53:44.6926.
mprotect(64 KiB, READ|WRITE), entered 1.9 ms earliermprotect(4 KiB, READ|EXEC), entered 0.5 ms earliermunmapof a 4 KiB RW view of the doublemapper memfd, entered 0.4 ms earliermembarrier. The lastmembarrier(PRIVATE_EXPEDITED)had returned 6.7 ms before the end.madvise. Everymadvisein the trace isMADV_DODUMP, issued right after an RWmprotectof the same range.mprotect: 887/s against 690 (334–1440)mprotectto EXEC: 118 against 68 (22–189)munmap: 357 against 188 (31–458)So strace does not single out a call. The hang comes in the ordinary stream of
mprotect/mmap/munmapfrom several processes at once.Two other results since the report:
DOTNET_GCName=libclrgc.so) 16 times in 965.This points below the runtime, at what the other online cores do while one process changes mappings. We are testing that on the kernel side now (idle cores kept polling with
nohlt). I have no runtime question at the moment, and will report back when that test has a result.The last 60 lines of the trace are below. The whole round is 248k lines, and I can share it if you want it.
last 60 lines