feat: add cached CUDA graph inference

[sj0618]

Summary

• add synchronous and asynchronous CUDA graph inference APIs
• cache captured graphs by batch size and device-pointer combination
• add an explicit cache-clear API for retiring device buffers
• expose CUDA graph benchmarking through --cuda-graph
• document graph lifetime and usage requirements

Closes #14

Verification

• cargo fmt --check
• git diff --check
• cargo check --all-targets could not run locally because CUDA/TensorRT and the required TENSORRT_LIBRARIES environment variable are unavailable on the development machine.

[sj0618]

@freeman94 Could you please review this PR when convenient? It implements the CUDA Graph support requested in #14. Local formatting checks pass; the remaining validation requires the CUDA/TensorRT environment.

[freeman94]

@freeman94 Could you please review this PR when convenient? It implements the CUDA Graph support requested in #14. Local formatting checks pass; the remaining validation requires the CUDA/TensorRT environment.

Hello, thank you for the PR! I will be on vacation until the start of August, so please bear with the delay in reviewing.

[freeman94]

Looks like you'll run into two issues that I encountered when I previously investigated implementing this feature. Both issues occur when using TensorRT in multithreaded contexts.

First, you'll get illegal access errors due to the default setting of global graph capture. See here.

Second, TensorRT seems to use synchronous aux streams by default. The fix there seems to be creating new streams manually and passing via setAuxStreams. See this and surrounding sections for more information.

Note, I've not tried any of these remediations and the nvidia docs read scattered and unclear about graph usage in heavily concurrent contexts, so not sure what will end up fixing it.