bug: DGX H100 Host with Alert: IB port(s) down: 1 of 9 ports are not active

[brusmith-nvidia]

Version

v0.10.6-0-ge9a595a23

Describe the bug.

trying to terminate host after released by Tenant. Carbide fails machine validation due to bad IB port, 1 out of 9.
But Host should only have 8 IB ports. Fix is to configure extra IB port to Ethernet port.
image showing error:

Minimum reproducible example

mlxconfig -d 29:00.0 set LINK_TYPE_P1=ETH

Relevant log output

Other/Misc.

No response

Code of Conduct

• I agree to follow NVIDIA Infra Controller's Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[RajMandaliya]

Looked into where the port count in this alert comes from (crates/health-report/src/lib.rs, ib_port_down) — total_ports is just whatever the health probe currently observes on the host, not a hardcoded per-SKU expectation. So the "9" here isn't a software bug, it's accurately reflecting that this specific host currently has 9 IB-capable ports, because the misconfigured port is presently reporting itself as InfiniBand rather than Ethernet. Once the mlxconfig fix is applied, that port should stop reporting as IB entirely and the count should correct itself to 8 with no code changes needed. Confirms this looks like a hardware config issue on this specific host, not something to fix in NICo itself. WIth that being said, I haven't tested it with any hardware, it's all from how I traced total_ports counts and code.

[ajf]

@hasayesh @nick-vetro I want to say this isn't a bug nor is it likely a FR. I remember us doing some work on this type of SKU a while back, but it's unique to 1 place. Can either of you make a recommendation here?