Issue with Pod Termination in InferenceService using nvcr.io/nim/meta/llama-3.1-8b-instruct:1.1.2 - No Terminate Signal Sent to Nim Server

[test-1pro]

Currently, I am using the nvcr.io/nim/meta/llama-3.1-8b-instruct:1.1.2 image to create an InferenceService following the provided guide, and the service is running properly. Pod creation and API calls are working fine, but I am encountering an issue when trying to delete the Pod.

It seems that the Terminate command is not being sent to the Nim server when I request the deletion of the InferenceService or Pod. There are no KILL signals in the internal logs either, and the Pod is only forcefully deleted when it reaches the terminationGracePeriodSeconds: 300.

Do I need to provide any additional options when starting the Nim server, or is this a known issue?

[bmcfeeters]

I'm seeing this same behavior when deploying using KServe. The scale up process works fine and is fast but, when no longer needed, the removal of the pod during scale down takes five minutes.

[prakashmirji]

@kylehh any update on this.

[kylehh]

This was marked as NAB internally because nimlib already handles SIGTERM in the code
So we have a reproduce step for the latest/newer version of NIMs?

[prakashmirji]

This was marked as NAB internally because nimlib already handles SIGTERM in the code So we have a reproduce step for the latest/newer version of NIMs?

@kylehh  which version nimlib has this fix? we are using NIM v1.3.2 ( pb24h2) 
When we try to delete the inferenceservice, NIM container is taking 5 mins to kill the child process.

in the NIM script, we have script
function terminate() {
  NIM_LLM_PID=$(pgrep -x "pt_main_thread")
  echo "Captured NIM_LLM_PID: ${NIM_LLM_PID:-0}"
  echo "$0 received SIGTERM, attempting graceful shutdown..."
  if [ -z "$NIM_LLM_PID" ]
  then
    echo "$0 has no NIM_LLM process to terminate"
    exit 0
  fi
  # Attempt graceful shutdown.
  echo "Attempting to shutdown process ${NIM_LLM_PID:-0}"
  kill -TERM "$NIM_LLM_PID"
  sleep 5 # Wait 5 seconds for graceful shutdown
  # Check if the child process is still running by sending signal 0.
  if kill -0 "$NIM_LLM_PID" 2>/dev/null; then
    echo "$0 graceful shutdown failed, forcefully killing process..."
    kill -9 "$NIM_LLM_PID"
  fi
  wait "$NIM_LLM_PID"
  exit 0
}
# Trap SIGINT or SIGTERM and call the terminate function.
trap terminate SIGINT SIGTERM

The container is not showing any echo statements output, this means, terminate() function not called , I think there is a bug here.

While waiting, ran this command on the nim container,

kubectl exec -it <pod> -n <namespace> -c kserve-container -- ps -e -o pid,ppid,cmd

it prints

PID 1     -> /bin/bash /opt/nim/start-server.sh
PID 54    -> /opt/nim/llm/.venv/bin/python3 -m nim_llm_sdk.entrypoints.openai
PID 125+  -> `orted` processes (likely related to NVIDIA’s use of MPI for GPU tasks)

It is not killing PID 54, waits for 5 mins. No logs in the NIM container.

nim script is listening for TRAP , but looking at the code, it is missing two things

python3 process is not starting in background,
wait “$child” statement is missing, so parent is not waiting for child process

Could you please take a look at it.

[jerryharrow]

Hmm. It does look like nvcr.io/nim/meta/llama-3.1-8b-instruct:1.8.5 has the proper handling, but
apparently the NIMs do not all the same wrapper infrastructure.

In nvcr.io/nim/nvidia/llama-3.2-nemoretriever-500m-rerank-v2 the problem happens above nimlib.

PID 1     -> /bin/bash /opt/nim/start-server.sh
PID 54    -> /opt/nim/llm/.venv/bin/python3 -m nim_llm_sdk.entrypoints.openai
PID 125+  -> `orted` processes (likely related to NVIDIA's use of MPI for GPU tasks)

The problem is the /opt/nim/start-server.sh doesn’t trap/forward the SIGTERM so nimlib never sees it.

/opt/nim/start_server.sh:

#!/usr/bin/env bash
set -eu

# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: LicenseRef-NvidiaProprietary

for file in /opt/nim/start_server.d/*; do
    if [[ "$file" == *.sh ]]; then
        . "$file"
    elif [[ "$file" == *.txt ]]; then
        cat "$file"
    fi
done

nemo-retriever-reranking-start-nim

There is another level above that (/opt/nvidia/nvidia_entrypoint.sh) -- but in that case it does an exec when starting /opt/nim/start-server.sh so there is not an additional shell process with a child.

[kylehh]

It's good to know that it's fixed in new version of LLM NIM and I will check w rerank team

[jerryharrow]

@kylehh while there was a fix made to /opt/nim/start-server.sh -- it doesn't seem to actually work as expected.
It looks like signals are blocked for some reason (perhaps by nim_llm_sdk.entrypoints.openai somehow? )

If you exec into the container and do 'kill -SIGTERM 1' it doesn't actually receive the signal (the terminate() function helpfully logs when it is invoked). In one test I was running the nim_llm_sdk.entrypoints.launch got killed by an OOM and in that case I could see that the SIGTERM I sent earlier actually eventually was delivered when process 55 exited.

INFO 2025-07-01 12:14:45.205 utils.py:374] Setting free_gpu_memory_fraction to 0.9
/opt/nim/start-server.sh: line 87:    55 Killed                  python3 -m nim_llm_sdk.entrypoints.launch
Captured NIM_LLM_PID: 0
/opt/nim/start-server.sh received SIGTERM, attempting graceful shutdown...

[kylehh]

@jerryharrow Yes, I can reprod the issue with kill -SIGTERM 1 in latest NIM v1.10.1
Killing the nim_llm_sdk.entrypoints.launch would trigger the terminate function instead.
So you observed kill -SIGTERM 1 for the /opt/nim/start-server.sh would never kill the process or take 10mins to kill?

[jerryharrow]

So you observed kill -SIGTERM 1 for the /opt/nim/start-server.sh would never kill the process or take 10mins to kill?

Exactly. It appears that the SIGTERM didn't get delivered until the python program completed.

[kylehh]

This bug was reported to engineer and a fix is in place for NIM 1.12.
I tested it and it can catch the SIGTERM as expected