Is your feature request related to a problem? Please describe.
cuDF Java can report a misleading error when its first CUDA call fails:
Caused by: ai.rapids.cudf.CudaFatalException: Fatal CUDA error encountered at: ../../src/main/native/src/CudaJni.cpp:56: 35 cudaErrorInsufficientDriver CUDA driver version is insufficient for CUDA runtime version
at ai.rapids.cudf.Cuda.memGetInfo(Native Method)
cudaErrorInsufficientDriver can mean that:
- no NVIDIA driver or GPU is present;
- no GPU was assigned to the container;
libcuda.so.1 is not mounted or discoverable; or
- the host driver is too old for the bundled CUDA runtime.
The current message suggests only the last cause. In Kubernetes, this can lead
operators to change the cuDF artifact when GPU provisioning is the real issue.
Describe the solution you'd like
For cudaErrorInsufficientDriver, append a non-throwing, best-effort diagnostic
to the original exception. Do not invoke external tools such as nvidia-smi.
The probe could run once per process and report:
- the CUDA runtime version;
- whether
libcuda.so.1 can be loaded;
- if available, driver initialization status, driver-supported CUDA version,
and visible device count.
The message should distinguish among an unavailable driver library, no visible
GPU, and a driver/runtime version mismatch. If the probe is inconclusive, list
these possible causes instead of stating that the driver is necessarily old.
Describe alternatives you've considered
Applications can run platform-specific checks, but nvidia-smi may not be on
PATH and device-file checks are not portable.
cudaDriverGetVersion() is useful, but CUDA documents that it may itself return
initialization errors. Checking whether libcuda.so.1 can be loaded provides
additional information without depending on an external executable.
At minimum, error 35 could list the likely causes even if cuDF cannot identify
which one occurred.
Additional context
The example originates from
CUDF_CUDA_TRY(cudaMemGetInfo(...)).
CUDA version API documentation:
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART____VERSION.html
Diagnostics should refer to the host driver's supported CUDA version, not the
host CUDA toolkit, because driver compatibility is the relevant boundary for a
bundled runtime.
Is your feature request related to a problem? Please describe.
cuDF Java can report a misleading error when its first CUDA call fails:
cudaErrorInsufficientDrivercan mean that:libcuda.so.1is not mounted or discoverable; orThe current message suggests only the last cause. In Kubernetes, this can lead
operators to change the cuDF artifact when GPU provisioning is the real issue.
Describe the solution you'd like
For
cudaErrorInsufficientDriver, append a non-throwing, best-effort diagnosticto the original exception. Do not invoke external tools such as
nvidia-smi.The probe could run once per process and report:
libcuda.so.1can be loaded;and visible device count.
The message should distinguish among an unavailable driver library, no visible
GPU, and a driver/runtime version mismatch. If the probe is inconclusive, list
these possible causes instead of stating that the driver is necessarily old.
Describe alternatives you've considered
Applications can run platform-specific checks, but
nvidia-smimay not be onPATHand device-file checks are not portable.cudaDriverGetVersion()is useful, but CUDA documents that it may itself returninitialization errors. Checking whether
libcuda.so.1can be loaded providesadditional information without depending on an external executable.
At minimum, error 35 could list the likely causes even if cuDF cannot identify
which one occurred.
Additional context
The example originates from
CUDF_CUDA_TRY(cudaMemGetInfo(...)).CUDA version API documentation:
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART____VERSION.html
Diagnostics should refer to the host driver's supported CUDA version, not the
host CUDA toolkit, because driver compatibility is the relevant boundary for a
bundled runtime.