Skip to content

[FEA] Improve diagnostics for CUDA initialization failures #24048

Description

@gerashegalov

Is your feature request related to a problem? Please describe.

cuDF Java can report a misleading error when its first CUDA call fails:

Caused by: ai.rapids.cudf.CudaFatalException: Fatal CUDA error encountered at: ../../src/main/native/src/CudaJni.cpp:56: 35 cudaErrorInsufficientDriver CUDA driver version is insufficient for CUDA runtime version
    at ai.rapids.cudf.Cuda.memGetInfo(Native Method)

cudaErrorInsufficientDriver can mean that:

  • no NVIDIA driver or GPU is present;
  • no GPU was assigned to the container;
  • libcuda.so.1 is not mounted or discoverable; or
  • the host driver is too old for the bundled CUDA runtime.

The current message suggests only the last cause. In Kubernetes, this can lead
operators to change the cuDF artifact when GPU provisioning is the real issue.

Describe the solution you'd like

For cudaErrorInsufficientDriver, append a non-throwing, best-effort diagnostic
to the original exception. Do not invoke external tools such as nvidia-smi.

The probe could run once per process and report:

  • the CUDA runtime version;
  • whether libcuda.so.1 can be loaded;
  • if available, driver initialization status, driver-supported CUDA version,
    and visible device count.

The message should distinguish among an unavailable driver library, no visible
GPU, and a driver/runtime version mismatch. If the probe is inconclusive, list
these possible causes instead of stating that the driver is necessarily old.

Describe alternatives you've considered

Applications can run platform-specific checks, but nvidia-smi may not be on
PATH and device-file checks are not portable.

cudaDriverGetVersion() is useful, but CUDA documents that it may itself return
initialization errors. Checking whether libcuda.so.1 can be loaded provides
additional information without depending on an external executable.

At minimum, error 35 could list the likely causes even if cuDF cannot identify
which one occurred.

Additional context

The example originates from
CUDF_CUDA_TRY(cudaMemGetInfo(...)).

CUDA version API documentation:
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART____VERSION.html

Diagnostics should refer to the host driver's supported CUDA version, not the
host CUDA toolkit, because driver compatibility is the relevant boundary for a
bundled runtime.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions