Skip to content

Whole-machine hang (CPU-complex watchdog reset) on NVIDIA Jetson AGX Xavier seconds after .NET processes start #137917

Open

Description

@peters

Description

On NVIDIA Jetson AGX Xavier (8 Carmel ARMv8.2 cores, Linux 5.10 from NVIDIA's L4T), the whole CPU complex stops within seconds of a .NET process starting. Nothing is printed on any console, including the serial console, and the board's hardware watchdog resets it about two minutes later (reset source BCCPLEXWDT). We see it on every L4T release we run (R35.1.0, R35.4.1, R35.6.5, R35.7.0 = JetPack 5.0.2 to 5.1.8), on NVIDIA's developer kit and on a third-party carrier board, in CI and on devices in the field.

We think the root cause is most likely below the OS (CPU or its firmware), so this is not a claim of a runtime bug. We are reporting it because a .NET process is involved in every freeze we have recorded, we have not been able to trigger it without .NET, and the first occurrence in our CI history coincides with a runtime update. We hope someone recognises the pattern, or can say which runtime behaviour could be switched off to avoid it.

Reproduction Steps

On an AGX Xavier with all 8 cores online, in a loop:

  1. sync; echo 3 > /proc/sys/vm/drop_caches (cold page cache)
  2. dotnet build tests/tests.csproj -c Release --no-restore --no-incremental -m:8 -nodeReuse:false -p:UseSharedCompilation=false on a small solution (8 class libraries of 200 trivial methods each plus one xUnit project referencing them; solution created with dotnet new by the same SDK)
  3. stop the build after 9 seconds (every freeze we recorded came within about 5 s of the build starting) and kill its processes

In a "bad" period this freezes the machine about once in 10 to 40 rounds. The rate changes over hours under the same load for reasons we do not understand, which makes A/B tests slow.

Expected behavior

The build runs or fails; the machine keeps running.

Actual behavior

The whole machine stops. A kernel module that samples each busy core's program counter every 100 µs into memory that survives the reset shows all busy cores stopping within about 0.1 ms of each other, each in ordinary user or kernel code (no common function, no cache maintenance in progress, sometimes GC running and sometimes not). Idle cores leave idle at the end. The kernel prints nothing: no soft lockup, no RCU stall, no panic.

Regression?

In our CI on two AGX Xavier runners, jobs began to be lost this way on 13 May 2026, the day the repositories moved from SDK 10.0.202 (runtime 10.0.6) to 10.0.300 (runtime 10.0.8) and started installing the SDK fresh for every job (so every job starts from a cold page cache). None were lost in the months before. That is a correlation, not proof: it also coincides with the cold-cache change.

A same-hours comparison on one machine (all cores, cold cache, SDKs in turn): SDK 10.0.202 0 freezes in 234 rounds, SDK 10.0.300 3 in 240, SDK 11.0.100 0 in 237. Too few freezes to be conclusive; a runtime-only swap (SDK 10.0.300 with runtime 10.0.6 against 10.0.8) is running now.

Known Workarounds

None that we trust. With CPUs 1 to 7 taken offline (CPU hotplug) it froze 0 times in 175 rounds where all cores froze 6 times in 103, but it has frozen once with one core online. Four cores online also froze.

Did not prevent it (each froze at least once with the setting):

  • DOTNET_EnableWriteXorExecute=0
  • DOTNET_TieredCompilation=0
  • DOTNET_gcConcurrent=0
  • DOTNET_UseGCWriteBarrierCopy=0
  • DOTNET_EnableArm64Atomics=0
  • DOTNET_ReadyToRun=0
  • an LD_PRELOAD hook around every __clear_cache call of libcoreclr
  • reading all SDK files into the page cache before the build
  • pinning the whole build to one core while the others stay online
  • kernel mitigations=off, nospectre_v2 nokaslr kpti=off, transparent_hugepage=never
  • fixed maximum CPU/EMC/GPU clocks, idle states disabled, CPU firmware background optimisation disabled

One freeze came 0.1 s after execve of SDK 11's dotnet build, inside the NativeAOT host, while the GC was committing its first regions (mprotect and madvise pairs 4 MiB apart), before any JIT.

Without .NET it never froze: programs that write and run millions of new functions a second on all cores, 89 million new executable pages, 620,000 process starts, cold starts of system binaries, CPU hotplug without .NET.

Configuration

  • NVIDIA Jetson AGX Xavier, L4T R35.1.0 to R35.7.0, Linux 5.10.x-tegra, Ubuntu 20.04 arm64
  • .NET SDK 10.0.202, 10.0.300, 11.0.100 (runtime 10.0.6, 10.0.8, 11.0.0), linux-arm64
  • Not specific to one carrier board

Other information

Questions we would be grateful for help with:

  • Did anything change between runtime 10.0.6 and 10.0.8 on linux-arm64 in how the runtime issues process-wide memory barriers (membarrier / FlushProcessWriteBuffers), cache maintenance, or GC region commit/decommit, that could be switched off with a setting?
  • Is a pattern like this known on other ARMv8 cores?

We can run any configuration or instrumentation you suggest.

Activity

  1. dotnet-policy-service commented on Nov 19, 2026

    @dotnet-policy-service
    Contributor

    Tagging subscribers to this area: @anicka-net, @dotnet/gc
    See info in area-owners.md if you want to be subscribed.

  2. janvorli commented on Nov 19, 2026

    @janvorli
    Member

    @peters I am not aware of anything that would change in the process wide memory barriers or cache maintenance. And nothing in the GC either.

    Can you try to run it under strace (with logging redirected to a file via the -o command line option) to see if it would reveal anything?

  3. peters commented on Nov 20, 2026

    @peters
    Author

    Thanks. Here is the strace result. This comment replaces my two earlier ones.

    Setup. strace -f -tt -s 48 on the whole dotnet build. The output is piped off the board over the network, because nothing written locally survives the reset.

    Result. The hang still occurs under strace: 2 hangs in 21 traced builds. The first trace ends about 1 ms before the hang. Its last line is at 01:53:44.6915, and our kernel recorder's last event is at 01:53:44.6926.

    • In flight at the end, in three different MSBuild worker processes:
      • mprotect(64 KiB, READ|WRITE), entered 1.9 ms earlier
      • mprotect(4 KiB, READ|EXEC), entered 0.5 ms earlier
      • munmap of a 4 KiB RW view of the doublemapper memfd, entered 0.4 ms earlier
    • membarrier. The last membarrier(PRIVATE_EXPEDITED) had returned 6.7 ms before the end.
    • madvise. Every madvise in the trace is MADV_DODUMP, issued right after an RW mprotect of the same range.
    • The final second is not unusual. Compared with the 24 seconds before it (median, min–max):
      • mprotect: 887/s against 690 (334–1440)
      • mprotect to EXEC: 118 against 68 (22–189)
      • munmap: 357 against 188 (31–458)
      • most memory calls in flight at once: 7 in the final second, 5–10 in the seconds before

    So strace does not single out a call. The hang comes in the ordinary stream of mprotect/mmap/munmap from several processes at once.

    Two other results since the report:

    • GC flavour makes no difference. Alternating round by round on two machines, the regions GC hung 14 times in 966 rounds and the segments GC (DOTNET_GCName=libclrgc.so) 16 times in 965.
    • The number of online cores does. With CPUs 1–7 offline: 0 hangs in 175 rounds, against 6 in 103 with all eight online, in alternating blocks. Pinning the build to CPU0 while the other cores stay online and idle does not help (2 in 74).

    This points below the runtime, at what the other online cores do while one process changes mappings. We are testing that on the kernel side now (idle cores kept polling with nohlt). I have no runtime question at the moment, and will report back when that test has a result.

    The last 60 lines of the trace are below. The whole round is 248k lines, and I can share it if you want it.

    last 60 lines
    13368 01:53:44.684946 futex(0xffe058000e80, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
    13413 01:53:44.685039 <... futex resumed>) = -1 ETIMEDOUT (Connection timed out)
    13492 01:53:44.685246 futex(0xaaab05184718, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
    13476 01:53:44.685336 madvise(0xfff0473f1000, 61440, MADV_DODUMP <unfinished ...>
    13475 01:53:44.685434 newfstatat(AT_FDCWD, "/var",  <unfinished ...>
    13413 01:53:44.685544 futex(0xffefe8003030, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
    13376 01:53:44.685642 sendto(66, "\10\302\0\0\0\7\0\0\0\1\0\0\0n\0\0\0\1\0\0\1\7MSBuild\350p\27\34\251F\337\10\1\0\0\0\f\0\0\0\1\4\0"..., 199, 0, NULL, 0 <unfinished ...>
    13368 01:53:44.685847 <... futex resumed>) = 1
    13130 01:53:44.685934 <... futex resumed>) = 0
    13492 01:53:44.686133 <... futex resumed>) = 1
    13480 01:53:44.686221 futex(0xffefc4000e4c, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
    13476 01:53:44.686308 <... madvise resumed>) = 0
    13475 01:53:44.686381 <... newfstatat resumed>{st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
    13413 01:53:44.686480 <... futex resumed>) = 0
    13376 01:53:44.686543 <... sendto resumed>) = 199
    13368 01:53:44.686631 epoll_pwait(197,  <unfinished ...>
    13184 01:53:44.686728 <... futex resumed>) = 0
    13480 01:53:44.686924 <... futex resumed>) = 1
    13475 01:53:44.686998 newfstatat(AT_FDCWD, "/var",  <unfinished ...>
    13437 01:53:44.687160 <... futex resumed>) = 0
    13413 01:53:44.687233 getrusage(RUSAGE_SELF,  <unfinished ...>
    13376 01:53:44.687320 sendto(66, "\10\302\0\0\0\7\0\0\0\0\0\0\0n\0\0\0\1\0\0\1\7MSBuild\350p\27\34\251F\337\10\1\0\0\0\f\0\0\0\1\4\0"..., 199, 0, NULL, 0 <unfinished ...>
    13368 01:53:44.687509 <... epoll_pwait resumed>[{EPOLLIN|EPOLLOUT, {u32=5, u64=5}}], 1024, -1, NULL, 8) = 1
    13184 01:53:44.687614 futex(0xaaab051846c0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
    13130 01:53:44.687708 futex(0xffe054000e00, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
    13484 01:53:44.687927 mprotect(0xffefd00a7000, 4096, PROT_READ|PROT_WRITE <unfinished ...>
    13480 01:53:44.688024 mprotect(0xffff54766000, 4096, PROT_READ|PROT_WRITE <unfinished ...>
    13475 01:53:44.688108 <... newfstatat resumed>{st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
    13413 01:53:44.688210 <... getrusage resumed>{ru_utime={tv_sec=3, tv_usec=765535}, ru_stime={tv_sec=0, tv_usec=989603}, ...}) = 0
    13376 01:53:44.688317 <... sendto resumed>) = 199
    13368 01:53:44.688395 epoll_pwait(197,  <unfinished ...>
    13184 01:53:44.688485 <... futex resumed>) = 0
    13130 01:53:44.688551 <... futex resumed>) = 1
    13480 01:53:44.688737 <... mprotect resumed>) = 0
    13477 01:53:44.688816 <... madvise resumed>) = 0
    13413 01:53:44.688894 futex(0xffefe800308c, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=431, tv_nsec=125335560}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
    13382 01:53:44.689003 <... futex resumed>) = 0
    13376 01:53:44.689083 futex(0xffefe401cdfc, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
    13368 01:53:44.689165 <... epoll_pwait resumed>[{EPOLLIN|EPOLLOUT, {u32=5, u64=5}}], 1024, -1, NULL, 8) = 1
    13110 01:53:44.689299 <... recvmsg resumed>{msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="ndowsOnly\4true\35LatestRuntimeFrameworkVersion\0053.0"..., iov_len=53979}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 53979
    13484 01:53:44.689564 <... mprotect resumed>) = 0
    13477 01:53:44.689636 mprotect(0xfff034612000, 65536, PROT_READ|PROT_WRITE <unfinished ...>
    13475 01:53:44.689720 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0x9af000 <unfinished ...>
    13437 01:53:44.689817 futex(0xffefc4000df0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
    13382 01:53:44.689907 futex(0xffe054000e00, FUTEX_WAIT_PRIVATE, 0, {tv_sec=20, tv_nsec=0} <unfinished ...>
    13368 01:53:44.690012 futex(0xffe058000e80, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
    13184 01:53:44.690101 futex(0xaaab0518471c, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=432, tv_nsec=626036830}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
    13130 01:53:44.690221 futex(0xffe058000e80, FUTEX_WAIT_PRIVATE, 0, {tv_sec=20, tv_nsec=0} <unfinished ...>
    13484 01:53:44.690437 mprotect(0xffefd00a8000, 4096, PROT_READ|PROT_WRITE <unfinished ...>
    13475 01:53:44.690524 <... mmap resumed>) = 0xffff8b265000
    13437 01:53:44.690598 <... futex resumed>) = 0
    13368 01:53:44.690681 <... futex resumed>) = 0
    13219 01:53:44.690748 <... futex resumed>) = -1 ETIMEDOUT (Connection timed out)
    13484 01:53:44.690954 <... mprotect resumed>) = 0
    13480 01:53:44.691031 mprotect(0xffff545a9000, 4096, PROT_READ|PROT_EXEC <unfinished ...>
    13475 01:53:44.691162 munmap(0xffff8b266000, 4096 <unfinished ...>
    13437 01:53:44.691247 futex(0xffefe401cdfc, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
    13368 01:53:44.691346 epoll_pwait(197,  <unfinished ...>
    13219 01:53:44.691462 futex(0xaaaaf628d660, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
    13130 01:53:44.691546 <... futex resumed>) = -1 EAGAIN (Resource temporarily unavailable)
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-GC-coreclruntriagedNew issue has not been triaged by the area owner

    Type

    No type

    Projects

    • Status
      No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions