fix: skip GPM/DCP metrics on fractional vGPU guests by kshitizlohia1994 · Pull Request #701 · NVIDIA/dcgm-exporter

kshitizlohia1994 · 2026-06-27T09:50:17Z

Problem

On GKE G4 nodes with fractional GPU allocations (g4-standard-6, g4-standard-12, g4-standard-24), dcgm-exporter emits repeated errors:

Got unexpected return -14 from m_gpmManager.GetLatestSample

and exports no profiling metrics. Full GPU shapes (g4-standard-48+) are unaffected.

The root cause is that fractional vGPU guests run under GPU_VIRTUALIZATION_MODE_VGPU, which prevents GPM (GPU Performance Monitoring) from collecting samples. The exporter attempted to collect DCP/profiling metrics unconditionally, resulting in a continuous error loop.

Fixes #661

Solution

Introduce internal/pkg/profiling with ValidateGPMSupport(), which runs before DCP metric collection on each reload cycle and:

Queries each GPU's virtualization mode via NVML GetVirtualizationMode — immediately returns unsupported for vGPU guest modes.
Queries GpmQueryDeviceSupport to check hardware-level GPM capability.
Performs a live two-sample GPM probe to confirm metrics can actually be retrieved at runtime.

When any GPU fails validation, DCP/profiling collection is disabled for that cycle while all standard DCGM metrics continue to be exported normally.

Test plan

Unit tests for VirtualizationModeBlocksGPM covering all virtualization mode variants
Verify on a full GPU shape (g4-standard-48+): DCP metrics continue to be collected
Verify on a fractional vGPU shape (g4-standard-6/12/24): no GPM errors, standard metrics still exported

Fractional vGPU shapes (e.g. GKE G4 g4-standard-6/12/24) run as vGPU guests and cannot provide GPM samples, causing dcgm-exporter to emit repeated "Got unexpected return -14 from m_gpmManager.GetLatestSample" errors and export no profiling metrics. Introduce internal/pkg/profiling with ValidateGPMSupport(), which checks each visible GPU for vGPU guest mode (via NVML GetVirtualizationMode) and performs a live GPM sample probe before collection begins. When any GPU fails validation, DCP/profiling metric collection is disabled for that reload cycle while standard DCGM metrics continue to be exported normally. Fixes NVIDIA#661

kshitizlohia1994 · 2026-07-03T16:56:38Z

@nccurry can you please review this pr when you get a chance?

rvatkar · 2026-07-08T21:41:19Z

 	}

+	if supported, reason := profiling.ValidateGPMSupport(); !supported {
+		config.CollectDCP = false


Is disabling DCP globally when any visible GPU fails validation the intended behavior? On a heterogeneous node, one unsupported or temporarily unhealthy GPU would remove profiling from all supported GPUs. Could we either make the decision per device/group or explicitly document and test this all-or-nothing policy?

changed to a per-gpu decision. profiling is only disabled globally when every gpm-capable gpu fails the probe and no other gpu can serve profiling, so one broken/unhealthy gpu no longer removes profiling from supported gpus.

rvatkar · 2026-07-08T21:41:49Z

+	"github.com/stretchr/testify/assert"
+)
+
+func TestVirtualizationModeBlocksGPM(t *testing.T) {


Could we add coverage for ValidateGPMSupport and its integration with queryDCPMetrics? At minimum, it would be useful to test pre-Hopper profiling, a supported vGPU, remote hostengine, mixed capabilities, the affected fractional-vGPU failure, and preservation of standard metrics when profiling is disabled.

added. table tests for ValidateGPMSupport (pre-hopper, healthy gpm, fractional-vgpu failure, mixed, remote hostengine, unknown) plus queryDCPMetrics integration covering CollectDCP/MetricGroups on keep vs disable.

Address review feedback on the fractional vGPU profiling fix: - Remove the blanket vGPU/HOST_VGPU virtualization-mode rejection and rely on a live GPM sample probe as the runtime signal, so supported MIG-backed vGPU profiles are no longer disabled. - Skip GPUs that do not support GPM (e.g. pre-Hopper A100) instead of treating them as unsupported; their DCGM_FI_PROF_* fields are served by the DCGM profiling module and are unaffected by GPM availability. - Trust DCGM's capability result under --remote-hostengine-info, since remote GPU UUIDs cannot be resolved via the local NVML library. - Decide per GPU: only disable profiling globally when every GPM-capable GPU fails the probe and no other GPU can serve profiling, so one broken or unhealthy GPU no longer removes profiling from supported GPUs. - Add table-driven tests covering pre-Hopper, healthy GPM, fractional vGPU failure, mixed capabilities, remote hostengine, and preservation of profiling on heterogeneous nodes.

Introduce test seams so the full GPM validation flow can be exercised without a real NVML library or GPU: - Extract a gpmDeviceChecker interface (init/shutdown/statusForUUID) behind a package-level var, moving all raw NVML calls into the production nvmlGPMChecker implementation. - Route queryDCPMetrics through a validateGPMSupportFn variable so tests can stub GPM validation. Add coverage for the previously untestable paths: - ValidateGPMSupport: remote hostengine short-circuit, NVML unavailable, device-count error, zero GPUs, single healthy, pre-Hopper (DCP module), single broken, homogeneous fractional vGPU, mixed healthy/broken, and device-info error handling. - queryDCPMetrics: metric-group query error, GPM-disable, and GPM-keep effects on CollectDCP and MetricGroups.

rvatkar · 2026-07-10T17:58:40Z

+	notApplicable := countStatus(statuses, gpmNotApplicable)
+	unknown := countStatus(statuses, gpmUnknown)
+
+	if broken > 0 && healthy == 0 && notApplicable == 0 && unknown == 0 {


Thanks for updating this. I think the mixed-GPU case still leaves the original error path active. The function returns only a global Boolean, so {healthy, broken} keeps profiling enabled for the shared GPU watch group. DCGM will therefore still call GPM for the broken GPU and emit the GetLatestSample error. Could we either exclude broken GPU IDs from profiling watches or use an explicitly documented all-or-nothing policy?

rvatkar · 2026-07-10T17:59:22Z

+			wantDisable: true,
+		},
+		{
+			name: "mixed healthy and broken GPM keeps profiling for healthy GPU",


Thanks for adding these tests. This case only asserts that profiling remains globally enabled; it does not prove that the broken GPU is excluded from profiling watches, so it passes with the current incomplete behavior. Could we add a test verifying that healthy GPUs retain profiling, broken GPUs are not watched for profiling fields, and standard metrics remain available when profiling is disabled?

rvatkar · 2026-07-10T18:11:41Z

+	defer deviceChecker.shutdown()
+
+	statuses := make([]gpuGPMStatus, 0, gpuCount)
+	for gpuID := uint(0); gpuID < gpuCount; gpuID++ {


Thanks for handling mixed capabilities. Could validation also respect config.GPUDeviceOptions? It currently evaluates every visible GPU, including unmonitored ones. For example, an unmonitored healthy GPU could keep profiling enabled for the only monitored—but broken—GPU. Could we base the decision on the entities that will actually be watched?

rvatkar · 2026-07-10T18:12:29Z

+		return gpmNotApplicable
+	}
+
+	if !probeGPMSample(device) {


Could we account for MIG entity sampling here? This probes the physical device with GpmSampleGet, while DCGM samples FE_GPU_I entities using GpmMigSampleGet with the GPU-instance ID. A whole-device probe therefore does not validate the path DCGM will use. Could we probe the actual monitored entity type, or conservatively skip MIG validation until that path is supported and tested?

rvatkar · 2026-07-10T18:13:01Z

+		return false
+	}
+
+	time.Sleep(gpmProbeSampleInterval)


Could we avoid sleeping once per GPM-capable GPU? Since validation is serial and runs synchronously during startup/topology reload, eight GPUs add at least four seconds of delay. One option would be to collect the first sample for every GPU, sleep once, then collect the second samples; bounded concurrency could also work.

rvatkar · 2026-07-10T18:13:38Z

+	assert.Equal(t, 0, countStatus(nil, gpmBroken))
+}
+
+// fakeChecker is a test double for gpmDeviceChecker that returns pre-programmed


Thanks for adding the checker seam. Could we move the seam closer to the NVML calls? The current fake returns a final status, so the production statusForUUID and probeGPMSample paths remain untested.
It would be useful to cover capability-query errors, both sample failures, metric-level errors, and sample cleanup on each failure path.

rvatkar reviewed Jul 8, 2026

View reviewed changes

Comment thread internal/pkg/profiling/gpm.go Outdated

rvatkar reviewed Jul 8, 2026

View reviewed changes

Comment thread internal/pkg/profiling/gpm.go Outdated

rvatkar reviewed Jul 8, 2026

View reviewed changes

Comment thread internal/pkg/profiling/gpm.go Outdated

rvatkar reviewed Jul 8, 2026

View reviewed changes

kshitizlohia1994 added 2 commits July 9, 2026 19:33

kshitizlohia1994 force-pushed the fix/gpm-fractional-vgpu-support branch from 75685e9 to 67ddf71 Compare July 9, 2026 14:04

rvatkar reviewed Jul 10, 2026

View reviewed changes

Uh oh!

Conversation

kshitizlohia1994 commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Test plan

Uh oh!

kshitizlohia1994 commented Jul 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kshitizlohia1994 commented Jun 27, 2026 •

edited

Loading