Hang at shutdown

[0-wiz-0]

Version

26.4.0

Platform

NetBSD 11.99.6/x86_64

Subsystem

No response

What steps will reproduce the bug?

When building chromium in pkgsrc, nodejs is used as a build tool. I have more than once seen a nodejs process hang during this.
Today I've accessed the hanging process and extracted a backtrace.

How often does it reproduce? Is there a required condition?

Twice in the last week (of about a chromium build a day).

What is the expected behavior? Why is that the expected behavior?

Clean shutdown.

What do you see instead?

The nodejs process does not exit. Here's the backtrace:

(gdb) thread apply all bt

Thread 3 (LWP 22392 of process 22392 "node-MainThread"):
#0  0x00007018c6e40dea in _lwp_wait () from /usr/lib/libc.so.12
#1  0x00007018c7275d88 in pthread_join (thread=0x7018c4b12000, valptr=valptr@entry=0x0) at /usr/src/lib/libpthread/pthread.c:718
#2  0x00007018c85c265e in uv_thread_join (tid=<optimized out>) at src/unix/thread.c:295
#3  0x00000000011cbd8a in node::WorkerThreadsTaskRunner::Shutdown (this=0x7018c4d168d0) at /usr/include/g++/bits/unique_ptr.h:193
#4  0x00000000011cccce in node::NodePlatform::Shutdown (this=0x7018c4cee340) at ../src/node_platform.cc:529
#5  node::NodePlatform::Shutdown (this=0x7018c4cee340) at ../src/node_platform.cc:526
#6  0x0000000001039db9 in node::V8Platform::Dispose (this=<optimized out>) at ../src/node_v8_platform-inl.h:95
#7  node::DisposePlatform () at ../src/node_v8_platform-inl.h:178
#8  node::DefaultProcessExitHandlerInternal (env=<optimized out>, exit_code=node::ExitCode::kNoFailure) at ../src/api/environment.cc:1155
#9  0x00000000010a58d2 in std::function<void(node::Environment*, node::ExitCode)>::operator() (this=0x7018c3d98bb8, __args#0=0x7018c3d98000, __args#1=node::ExitCode::kNoFailure) at /usr/include/g++/bits/std_function.h:59
1
#10 node::Environment::Exit (this=0x7018c3d98000, exit_code=node::ExitCode::kNoFailure) at ../src/env.cc:1930
#11 0x00007f7ff774fae9 in ?? ()
#12 0x00007018c3d464e9 in ?? ()
#13 0x0000000000000001 in ?? ()
#14 0x00007f7fff8bbc90 in ?? ()
#15 0x0000000000000024 in ?? ()
#16 0x00007f7fff8bbd48 in ?? ()
#17 0x00007f7ff774e0a4 in ?? ()
#18 0x00007018c4a04000 in ?? ()
#19 0x00007018c3f00011 in ?? ()
#20 0x00007018c282c181 in ?? ()
#21 0x00007018c28061a1 in ?? ()
#22 0x00007018c2838339 in ?? ()
#23 0x0000000000000000 in ?? ()


Thread 2 (LWP 6025 of process 22392 "node-V8Worker"):
#0  0x00007018c6eb5a6a in ___lwp_park60 () from /usr/lib/libc.so.12
#1  0x00007018c7273d3d in pthread_cond_timedwait (cond=0x7018c4c47770, mutex=0x7018c4c47740, abstime=0x0) at /usr/src/lib/libpthread/pthread_cond.c:178
#2  0x00000000026b24e3 in absl::synchronization_internal::PthreadWaiter::Wait (this=this@entry=0x7018c4c47740, t=..., t@entry=...) at ../deps/v8/third_party/abseil-cpp/absl/synchronization/internal/pthread_waiter.cc:117
#3  0x00000000026b22a9 in AbslInternalPerThreadSemWait (t=...) at ../deps/v8/third_party/abseil-cpp/absl/synchronization/internal/per_thread_sem.cc:93
#4  0x000000000269f7b3 in absl::CondVar::WaitCommon (this=this@entry=0x7018c4ca84d0, mutex=mutex@entry=0x7018c4ca84c8, t=..., t@entry=...) at ../deps/v8/third_party/abseil-cpp/absl/synchronization/mutex.cc:2641
#5  0x00000000026d4c89 in absl::CondVar::Wait (this=0x7018c4ca84d0, mu=0x7018c4ca84c8) at ../deps/v8/third_party/abseil-cpp/absl/synchronization/mutex.h:989
#6  0x0000000001a9f776 in operator() (__closure=<synthetic pointer>) at ../deps/v8/src/heap/collection-barrier.cc:166
#7  v8::internal::LocalHeap::ParkAndExecuteCallback<v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()> > (this=0x7018c2dcf688, callback=...) at
 ../deps/v8/src/heap/local-heap-inl.h:71
#8  operator() (__closure=<optimized out>) at ../deps/v8/src/heap/local-heap-inl.h:88
#9  heap::base::Stack::SetMarkerForBackgroundThreadAndCallbackImpl<v8::internal::LocalHeap::ExecuteWhileParked<v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()> >(v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()>)::<lambda()> >(heap::base::Stack *, void *, const void *) (stack=0x7018c4ca4760, argument=<optimized out>, stack_end=<optimized out>) at ../deps/v8/src/heap/base/stack.h:208
#10 0x00000000013d0f59 in PushAllRegistersAndIterateStack ()
#11 0x0000000001aa0085 in heap::base::Stack::SetMarkerForBackgroundThreadAndCallback<v8::internal::LocalHeap::ExecuteWhileParked<v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()> >(v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()>)::<lambda()> > (this=0x7018c4ca4760, thread=<optimized out>, callback=...) at ../deps/v8/src/heap/base/stack.h:90
#12 v8::internal::LocalHeap::ExecuteWithStackMarker<v8::internal::LocalHeap::ExecuteWhileParked<v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda
()> >(v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()>)::<lambda()> > (this=0x7018c2dcf688, callback=...) at ../deps/v8/src/heap/local-heap-i
nl.h:80
#13 v8::internal::LocalHeap::ExecuteWhileParked<v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()> > (this=0x7018c2dcf688, callback=...) at ../deps/v8/src/heap/local-heap-inl.h:87
#14 v8::internal::CollectionBarrier::AwaitCollectionBackground (this=0x7018c4ca84c0, local_heap=0x7018c2dcf688, kind=<optimized out>) at ../deps/v8/src/heap/collection-barrier.cc:158
#15 0x0000000001b24475 in v8::internal::HeapAllocator::CollectGarbageAndRetryAllocation (this=this@entry=0x7018c2dcf730, allocate=..., allocation=allocation@entry=v8::internal::AllocationType::kCode) at ../deps/v8/src/heap/heap-allocator.cc:543
#16 0x0000000001b24664 in v8::internal::HeapAllocator::RetryCustomAllocateLight (this=0x7018c2dcf730, allocate=..., allocation=v8::internal::AllocationType::kCode) at ../deps/v8/src/heap/heap-allocator.cc:518
#17 0x0000000001b24721 in v8::internal::HeapAllocator::AllocateRawSlowPath (this=this@entry=0x7018c2dcf730, retry_mode=retry_mode@entry=v8::internal::HeapAllocator::kLightRetry, size=size@entry=1664, allocation=allocation@entry=v8::internal::AllocationType::kCode, origin=origin@entry=v8::internal::AllocationOrigin::kRuntime, alignment=alignment@entry=v8::internal::kTaggedAligned, hint=hint@entry=...) at ../deps/v8/src/heap/heap-allocato
r.cc:215
#18 0x0000000001afa11a in v8::internal::HeapAllocator::AllocateRawWith<(v8::internal::HeapAllocator::AllocationRetryMode)0> (this=0x7018c2dcf730, size=1664, allocation=v8::internal::AllocationType::kCode, origin=v8::internal::AllocationOrigin::kRuntime, alignment=v8::internal::kTaggedAligned, hint=...) at ../deps/v8/src/heap/allocation-result.h:41
#19 v8::internal::LocalHeap::AllocateRawWith<(v8::internal::HeapAllocator::AllocationRetryMode)0> (this=0x7018c2dcf688, object_size=1664, type=v8::internal::AllocationType::kCode, origin=v8::internal::AllocationOrigin::k
Runtime, alignment=v8::internal::kTaggedAligned, hint=...) at ../deps/v8/src/heap/local-heap-inl.h:44
#20 v8::internal::Factory::CodeBuilder::AllocateUninitializedInstructionStream (this=this@entry=0x7018c2dcf280, retry_allocation_or_fail=retry_allocation_or_fail@entry=false) at ../deps/v8/src/heap/factory.cc:285
#21 0x0000000001b12c6a in v8::internal::Factory::CodeBuilder::BuildInternal (this=this@entry=0x7018c2dcf280, retry_allocation_or_fail=retry_allocation_or_fail@entry=false) at ../deps/v8/src/heap/factory.cc:150
#22 0x0000000001b1348a in v8::internal::Factory::CodeBuilder::TryBuild (this=this@entry=0x7018c2dcf280) at ../deps/v8/src/heap/factory.cc:295
#23 0x00000000020f58bd in v8::internal::baseline::BaselineCompiler::Build (this=this@entry=0x7018c2dcf3c0) at ../deps/v8/src/baseline/baseline-compiler.cc:359
#24 0x00000000020f25af in v8::internal::baseline::BaselineCompilerTask::Compile (this=0x7018a798a800, local_isolate=0x7018c2dcf680) at ../deps/v8/src/execution/local-isolate.h:61
#25 v8::internal::baseline::BaselineBatchCompilerJob::Compile (this=0x7018c3f53ce0, local_isolate=0x7018c2dcf680) at ../deps/v8/src/baseline/baseline-batch-compiler.cc:129
#26 v8::internal::baseline::ConcurrentBaselineCompiler::JobDispatcher::Run (this=<optimized out>, delegate=<optimized out>) at ../deps/v8/src/baseline/baseline-batch-compiler.cc:171
#27 0x00000000025f1f71 in v8::platform::DefaultJobWorker::Run (this=0x7018c3f53180) at ../deps/v8/src/libplatform/default-job.h:147
#28 0x00000000011cc407 in node::(anonymous namespace)::PlatformWorkerThread (data=0x7018c4cd7820) at ../src/node_platform.cc:80
#29 0x00007018c7275383 in pthread__create_tramp (cookie=0x7018c4b12000) at /usr/src/lib/libpthread/pthread.c:601
#30 0x00007018c6e96de0 in ?? () from /usr/lib/libc.so.12
#31 0x0000000000000000 in ?? ()

Thread 1 (LWP 10116 of process 22392 "SignalInspector"):
#0  0x00007018c6e40f6a in _ksem_wait () from /usr/lib/libc.so.12
#1  0x00007018c85c2cc4 in uv__sem_wait (sem=0x491bf80 <node::inspector::(anonymous namespace)::start_io_thread_semaphore>) at src/unix/thread.c:660
#2  uv_sem_wait (sem=0x491bf80 <node::inspector::(anonymous namespace)::start_io_thread_semaphore>) at src/unix/thread.c:716
#3  0x00000000012f9dff in node::inspector::(anonymous namespace)::StartIoThreadMain (unused=<optimized out>) at ../src/inspector_agent.cc:94
#4  0x00007018c7275383 in pthread__create_tramp (cookie=0x7018c4b15800) at /usr/src/lib/libpthread/pthread.c:601
#5  0x00007018c6e96de0 in ?? () from /usr/lib/libc.so.12
#6  0x0000000000000000 in ?? ()

It looks like one thread is calling pthread_join, while the thread it wants to join is hanging in an unrestricted pthread_cond_timedwait (abstime = 0x0).

Probably the shutdown needs to first wake up the waiting thread.

Additional information

No response

[dominexmacedon-docs]

I’ve hit the same hang when building Chromium with Node.js on NetBSD. The backtrace shows one thread stuck in pthread_cond_timedwait with abstime = 0, while another thread is trying to pthread_join it during shutdown:

WorkerThreadsTaskRunner::Shutdown calls uv_thread_join

The joined thread is parked inside V8’s CollectionBarrier::AwaitCollectionBackground

On NetBSD, pthread_cond_timedwait with a null abstime doesn’t return immediately, so the thread just waits forever

That leaves Node’s shutdown blocked because the waiting thread never gets signaled. Other platforms may behave differently, but on NetBSD this effectively means the process won’t exit cleanly.

It looks like the shutdown sequence needs to explicitly wake up or signal the condition variable before joining these worker threads. Without that, the join call will hang indefinitely. This is reproducible in pkgsrc builds and has been seen multiple times in the last week.