Skip to content

feat(helm): machine-a-tron mac address allocation #3700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
akorobkov-nvda wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(helm): machine-a-tron mac address allocation #3700

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 23 additions & 7 deletions docs/development/machine-a-tron-deployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,13 +330,29 @@ seeds, SPIFFE URI). Multi-pod adds the following requirements, all hit in practi
`bmc_proxy` the Redfish client sends no `Forwarded` header; older bmc-mock
builds then 404 every request with `no router configured`. Leave
`site_explorer.bmc_proxy` unset in this mode.
1. **Disjoint MAC pools per pod.** All machine-a-tron instances derive MACs
from the same default pools, so the pod that leases second is rejected on
every DHCP with `Network segment mismatch for existing MAC address` and
simulates nothing. Until the chart grows a per-pod MAC knob, supply full
per-pod TOML overrides via `configFiles.matConfigs.<pod>` setting distinct
`mac_address_pool` / `hw_mac_address_ranges` bases (see the example
values file).
1. **Disjoint MAC pools per pod.** The Helm chart **auto-generates** unique
MAC address pools per pod based on pod index. The format is
`02:00:PP:XX:XX:XX` where `PP` is the pod index (0x00, 0x01, etc.).

Example with 3 pods:

- Pod mat-0: `02:00:00:00:00:00` (base)
- Pod mat-1: `02:00:01:00:00:00` (base)
- Pod mat-2: `02:00:02:00:00:00` (base)

To override, set per-pod MAC pools in values:

```yaml
pods:
mat-0:
macAddressPool:
base: "02:00:00:00:00:00"
hostBits: 20
```

Or disable auto-generation with `macAddressPool.enabled: false` and use
full TOML overrides via `configFiles.matConfigs.<pod>`.

1. **One NICo network segment per pod CIDR.** `network_prefixes` allows one
IPv4 prefix per segment, so each pod CIDR needs its own cloned underlay
segment (same technique as the scale-mode segment fallback), gateway `.1`,
Expand Down
134 changes: 21 additions & 113 deletions helm-prereqs/values/machine-a-tron-multipod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# machine-a-tron site values — MULTI-POD bmcServices mode (validated example)
# machine-a-tron site values — MULTI-POD bmcServices mode
#
# Two machine-a-tron pods, each simulating 5 hosts x 2 DPUs (30 BMCs total),
# each with a dedicated per-BMC ClusterIP Service range. Copy this file and
Expand All @@ -11,47 +11,24 @@
#
# * Kubernetes 1.29+ for the ServiceCIDR object (beta 1.31+). On OLDER
# clusters set bmcServices.serviceCIDR.create=false and pick pod CIDRs
# INSIDE the apiserver's --service-cluster-ip-range — any clusterIP outside
# that range is rejected outright ("the provided IP is not in the valid
# range"). Verify the chosen sub-ranges are free of existing ClusterIPs:
# kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}'
# INSIDE the apiserver's --service-cluster-ip-range.
#
# * The machine-a-tron image must include the bmc-mock Host-header routing
# fallback (PR #3190). Without a bmc_proxy the Redfish client sends no
# "Forwarded" header, and older mocks 404 every request with
# "no router configured". Do NOT set site_explorer.bmc_proxy in this mode —
# fallback (PR #3190). Do NOT set site_explorer.bmc_proxy in this mode —
# NICo dials each BMC's ClusterIP directly.
#
# * Every pod MUST get disjoint MAC pools (the per-pod TOML overrides below).
# All machine-a-tron instances otherwise derive identical MACs from the
# default pools (02:00:00:00:00:00 / 06:00:00:00:00:00); the pod that
# leases second is rejected on every DHCP with "Network segment mismatch
# for existing MAC address" and simulates nothing. The chart has no
# per-pod MAC knob yet, so the configFiles.matConfigs overrides are the
# only way to set mac_address_pool / hw_mac_address_ranges per pod.
# * MAC address pools are AUTO-GENERATED per pod by the Helm chart:
# - mac_address_pool: 02:00:PP:00:00:00 (PP = pod index)
# - hw_mac_address_ranges: 02:01:PP:00:00:00 (PP = pod index)
# No manual TOML overrides needed.
#
# * One NICo network segment per pod CIDR: network_prefixes has a
# one-IPv4-prefix-per-segment constraint, so the pod CIDRs cannot be added
# as two prefixes of one OOB segment. Clone one underlay segment per pod
# (same jsonb_populate_record technique as setup-machine-a-tron.sh scale
# mode), prefix = the pod CIDR, gateway = .1, num_reserved = 1 — the chart
# allocates BMC Service IPs starting at .2, which then lines up with the
# DHCP allocator.
# one-IPv4-prefix-per-segment constraint, so each pod CIDR needs its own
# cloned underlay segment with gateway = .1, num_reserved = 1.
#
# * Vendor-specific factory credential path: hw types whose Redfish vendor
# string is not recognized by libredfish (e.g. wiwynn_gb200_nvl reports
# "WIWYNN") resolve to BMCVendor::Unknown, so the host factory credential
# must be seeded at machines/all_hosts/factory_default/bmc-metadata-items/unknown.
#
# * GB-class hosts (NvidiaGBx00 flow, e.g. wiwynn_gb200_nvl) require
# expected_machines rows with a WORKING BMC credential before exploration
# completes (error NICO-SITEEXPLORER-141 "Missing credential
# expected_machine"). machine-a-tron registers them itself when
# registerExpectedMachines=true, but only on nico-api builds with the
# Machineatron AddExpectedMachine RBAC grant — on older builds the call is
# 403'd and the rows must be inserted directly (see the fallback in
# setup-machine-a-tron.sh; the row password must be the PINNED password
# below, not the factory default). DPU BMCs explore without expected rows.
# * GB-class hosts (NvidiaGBx00 flow) require expected_machines rows with a
# WORKING BMC credential before exploration completes. machine-a-tron
# registers them when registerExpectedMachines=true.

certificate:
uris:
Expand All @@ -65,9 +42,6 @@ global:
imagePullSecrets:
- name: machine-a-tron-pull

# Build and push the image to the registry of your choice (see the deployment
# guide); the setup script injects these from NICO_IMAGE_REGISTRY/MAT_IMAGE_TAG
# when deploying via setup-machine-a-tron.sh.
image:
pullPolicy: IfNotPresent
repository: ""
Expand All @@ -86,12 +60,15 @@ bmcServices:
servicePort: 443
serviceCIDR:
# Requires Kubernetes 1.29+. On older clusters leave false and carve the
# pod CIDRs from the cluster's existing service range (see header).
# pod CIDRs from the cluster's existing service range.
create: false

pods:
mat-0:
cidr: "10.233.58.0/23"
# Auto-generated MAC pools:
# mac_address_pool: 02:00:00:00:00:00
# hw_mac_address_ranges: 02:01:00:00:00:00
machines:
compute:
hwType: wiwynn_gb200_nvl
Expand All @@ -101,6 +78,9 @@ pods:
adminDhcpRelayAddress: "10.102.0.1"
mat-1:
cidr: "10.233.60.0/23"
# Auto-generated MAC pools:
# mac_address_pool: 02:00:01:00:00:00
# hw_mac_address_ranges: 02:01:01:00:00:00
machines:
compute:
hwType: wiwynn_gb200_nvl
Expand All @@ -117,78 +97,6 @@ resources:
cpu: 500m
memory: 512Mi

# Full per-pod TOML overrides. The chart uses these VERBATIM instead of
# generating mat.toml from pods.<name>.machines, so they must repeat the
# machine sections — keep them in sync with the pods block above. They exist
# solely to give each pod disjoint MAC pools (see header).
# NOTE: Only use configFiles.matConfigs for advanced TOML overrides not exposed in values.
configFiles:
matConfigs:
mat-0: |
# Pod: mat-0 CIDR: 10.233.58.0/23
carbide_api_url = "https://nico-api.nico-system.svc.cluster.local:1079"
interface = "NOTUSED"
tui_enabled = false
bmc_mock_port = 1266
use_single_bmc_mock = true
mock_bmc_ssh_server = true
mock_bmc_ssh_port = 2222
cleanup_on_quit = false
register_expected_machines = true
host_bmc_password = "NicoSiteRoot1"
dpu_bmc_password = "NicoSiteRoot1"
api_refresh_interval = "2s"

mac_address_pool = { base = "02:00:00:00:00:00", host_bits = 24 }
hw_mac_address_ranges = { base = "06:00:00:00:00:00", host_bits = 32, range_host_bits = 8 }

[machines.compute]
hw_type = "wiwynn_gb200_nvl"
host_count = 5
vpc_count = 0
subnets_per_vpc = 0
dpu_per_host_count = 2
dpu_reboot_delay = 1
host_reboot_delay = 1
scout_run_interval = "60s"
run_interval_working = "1s"
run_interval_idle = "10s"
network_status_run_interval = "20s"
template_dir = "/opt/machine-a-tron/templates"
oob_dhcp_relay_address = "10.233.58.1"
admin_dhcp_relay_address = "10.102.0.1"
dpus_in_nic_mode = false
mat-1: |
# Pod: mat-1 CIDR: 10.233.60.0/23
carbide_api_url = "https://nico-api.nico-system.svc.cluster.local:1079"
interface = "NOTUSED"
tui_enabled = false
bmc_mock_port = 1266
use_single_bmc_mock = true
mock_bmc_ssh_server = true
mock_bmc_ssh_port = 2222
cleanup_on_quit = false
register_expected_machines = true
host_bmc_password = "NicoSiteRoot1"
dpu_bmc_password = "NicoSiteRoot1"
api_refresh_interval = "2s"

# Disjoint MAC spaces from mat-0 (outside its 2^24 / 2^32 host spans)
mac_address_pool = { base = "02:00:01:00:00:00", host_bits = 24 }
hw_mac_address_ranges = { base = "06:01:00:00:00:00", host_bits = 32, range_host_bits = 8 }

[machines.compute]
hw_type = "wiwynn_gb200_nvl"
host_count = 5
vpc_count = 0
subnets_per_vpc = 0
dpu_per_host_count = 2
dpu_reboot_delay = 1
host_reboot_delay = 1
scout_run_interval = "60s"
run_interval_working = "1s"
run_interval_idle = "10s"
network_status_run_interval = "20s"
template_dir = "/opt/machine-a-tron/templates"
oob_dhcp_relay_address = "10.233.60.1"
admin_dhcp_relay_address = "10.102.0.1"
dpus_in_nic_mode = false
matConfigs: {}
78 changes: 71 additions & 7 deletions helm/charts/nico-machine-a-tron/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ Helm chart for deploying Machine-A-Tron - a mock machine simulator for NICo test

## Overview

Machine-A-Tron creates simulated bare-metal machines that behave like real hosts, allowing you to:
Machine-A-Tron creates simulated bare-metal machines that behave like real hosts,
allowing you to:

- Test NICo without physical hardware
- Simulate multiple hosts, DPUs, switches and power shelves
Expand Down Expand Up @@ -42,13 +43,15 @@ helm upgrade --install nico ./helm \
[site_explorer]
override_target_host = "nico-machine-a-tron-bmc-mock"
override_target_port = 1266
```

```text

---

## Mode 2: ClusterIP Mode (Scale Testing)
Comment on lines +47 to 51

@coderabbitai coderabbitai Bot Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Restore code-fence boundaries around the TOML examples. Both fences begin after their intended snippets, causing following sections to render as literal text.

  • helm/charts/nico-machine-a-tron/README.md#L47-L51: open a toml fence before [site_explorer] and close it after override_target_port.
  • helm/charts/nico-machine-a-tron/README.md#L235-L236: open a toml fence before [networks.pod-0-oob] and close it after the final gateway.

As per path instructions, review Markdown for correctness and operator usability.

📍 Affects 1 file
  • helm/charts/nico-machine-a-tron/README.md#L47-L51 (this comment)
  • helm/charts/nico-machine-a-tron/README.md#L235-L236
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@helm/charts/nico-machine-a-tron/README.md` around lines 47 - 51, Restore the
TOML code-fence boundaries in helm/charts/nico-machine-a-tron/README.md: at
lines 47-51, open a toml fence before [site_explorer] and close it after
override_target_port; at lines 235-236, open a toml fence before
[networks.pod-0-oob] and close it after the final gateway. Ensure subsequent
Markdown sections render normally.

Source: Path instructions


**Use for load testing environments where simulated machines run alongside real hardware.**
**Use for load testing environments where simulated machines run alongside real
hardware.**

Each simulated BMC gets a dedicated ClusterIP service. Supports multi-pod deployments.

Expand Down Expand Up @@ -167,6 +170,7 @@ pods:
### What Gets Created

**Per pod:**

| Resource | Name Pattern |
|----------|--------------|
| Deployment | `nico-machine-a-tron-pod-0` |
Expand All @@ -175,18 +179,21 @@ pods:
| Service | `nico-machine-a-tron-pod-0-bmc-mock` |

**Per BMC:**

| Resource | Name Pattern |
|----------|--------------|
| ClusterIP Service | `nico-machine-a-tron-bmc-10-100-0-2` |

**Cluster-scoped:**

| Resource | Name |
|----------|------|
| ServiceCIDR | `nico-machine-a-tron-bmc-cidr` |

### Scale Guidelines

**BMC count per GB200 NVL72 rack:**

| Component | Count | BMCs per Unit | Total |
|-----------|-------|---------------|-------|
| Compute trays | 18 | 3 (1 tray + 2 BF3) | 54 |
Expand All @@ -195,6 +202,7 @@ pods:
| **Total per rack** | | | **71** |

**CIDR sizing:**

| CIDR | Usable IPs | Racks per Pod |
|------|------------|---------------|
| /24 | 253 | ~3 |
Expand Down Expand Up @@ -224,7 +232,8 @@ gateway = "10.100.0.1"
type = "underlay"
prefix = "10.100.4.0/22"
gateway = "10.100.4.1"
```

```text

---

Expand All @@ -236,6 +245,9 @@ gateway = "10.100.4.1"
pods:
<pod-name>:
cidr: "" # Required for bmcServices mode
macAddressPool: # Optional: override auto-generated MAC pool
base: "02:00:00:00:00:00"
hostBits: 16
machines:
<group-name>:
hwType: wiwynn_gb200_nvl
Expand All @@ -246,6 +258,55 @@ pods:
# ... other machine settings
```

### MAC Address Pool Configuration

Each pod needs unique MAC addresses to avoid collisions in multi-pod deployments.
By default, the chart auto-generates unique MAC pools per pod based on pod index.

**Two pools are configured:**

1. **`mac_address_pool`** - Standalone MACs for BMC and management interfaces:
- Host BMC MAC
- DPU BMC MAC (per DPU)
- DPU Host MAC (representor interface, per DPU)
- DPU OOB MAC (out-of-band management, per DPU)
- NVOS MACs (for switches)

*Example: GB200 host with 2 DPUs needs ~7 MACs from this pool.*

2. **`hw_mac_address_ranges`** - Contiguous MAC blocks for NICs:
- ConnectX NICs (e.g., CX-8 needs 10 consecutive MACs)
- Storage NICs
- Management NICs

*Real hardware has contiguous MAC blocks assigned at manufacturing.*

```yaml
macAddressPool:
enabled: true # Enable auto-generation
basePrefix: "02:00" # First 2 bytes (locally administered)
hostBits: 16 # ~65K MACs per pod

hwMacAddressRanges:
enabled: true # Enable auto-generation
basePrefix: "02:01" # First 2 bytes
hostBits: 24 # Total range size
rangeHostBits: 8 # 256 MACs per allocation
```

**Auto-generated formats:**

- `mac_address_pool`: `02:00:PP:XX:XX:XX` (PP = pod index)
- `hw_mac_address_ranges`: `02:01:PP:XX:XX:XX` (PP = pod index)

**Example with 3 pods:**

| Pod | mac_address_pool | hw_mac_address_ranges |
|-----|------------------|----------------------|
| mat-0 | `02:00:00:00:00:00` | `02:01:00:00:00:00` |
| mat-1 | `02:00:01:00:00:00` | `02:01:01:00:00:00` |
| mat-2 | `02:00:02:00:00:00` | `02:01:02:00:00:00` |

### BMC Services Configuration

```yaml
Expand Down Expand Up @@ -303,12 +364,15 @@ Causes:

Verify selector labels match:

# Check service selector
## Check service selector

kubectl -n nico-mat get svc nico-machine-a-tron-bmc-10-100-0-2 -o jsonpath='{.spec.selector}'

# Check pod labels
## Check pod labels

kubectl -n nico-mat get pods -l nvidia-infra-controller/pod-name=pod-0 --show-labels
```

```text
Comment on lines +367 to +375

@coderabbitai coderabbitai Bot Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Restore troubleshooting heading and command-block structure.

These ## headings escape the parent troubleshooting section, and the fence opens after both commands. Use lower-level headings and closed bash fences around each command.

As per path instructions, review Markdown for correctness and operator usability.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@helm/charts/nico-machine-a-tron/README.md` around lines 367 - 375, Update the
troubleshooting section in the README by changing the “Check service selector”
and “Check pod labels” headings to lower-level headings that remain nested under
the parent section. Wrap each kubectl command in its own closed bash code fence,
keeping the existing commands and making the Markdown structure render
correctly.

Source: Path instructions


### View Generated Config

Expand Down
Loading
Loading