diff --git a/docs/development/machine-a-tron-deployment.md b/docs/development/machine-a-tron-deployment.md index 3ffebba58c..249a38e0e3 100644 --- a/docs/development/machine-a-tron-deployment.md +++ b/docs/development/machine-a-tron-deployment.md @@ -330,13 +330,29 @@ seeds, SPIFFE URI). Multi-pod adds the following requirements, all hit in practi `bmc_proxy` the Redfish client sends no `Forwarded` header; older bmc-mock builds then 404 every request with `no router configured`. Leave `site_explorer.bmc_proxy` unset in this mode. -1. **Disjoint MAC pools per pod.** All machine-a-tron instances derive MACs - from the same default pools, so the pod that leases second is rejected on - every DHCP with `Network segment mismatch for existing MAC address` and - simulates nothing. Until the chart grows a per-pod MAC knob, supply full - per-pod TOML overrides via `configFiles.matConfigs.` setting distinct - `mac_address_pool` / `hw_mac_address_ranges` bases (see the example - values file). +1. **Disjoint MAC pools per pod.** The Helm chart **auto-generates** unique + MAC address pools per pod based on pod index. The format is + `02:00:PP:XX:XX:XX` where `PP` is the pod index (0x00, 0x01, etc.). + + Example with 3 pods: + + - Pod mat-0: `02:00:00:00:00:00` (base) + - Pod mat-1: `02:00:01:00:00:00` (base) + - Pod mat-2: `02:00:02:00:00:00` (base) + + To override, set per-pod MAC pools in values: + + ```yaml + pods: + mat-0: + macAddressPool: + base: "02:00:00:00:00:00" + hostBits: 20 + ``` + + Or disable auto-generation with `macAddressPool.enabled: false` and use + full TOML overrides via `configFiles.matConfigs.`. + 1. **One NICo network segment per pod CIDR.** `network_prefixes` allows one IPv4 prefix per segment, so each pod CIDR needs its own cloned underlay segment (same technique as the scale-mode segment fallback), gateway `.1`, diff --git a/helm-prereqs/values/machine-a-tron-multipod.yaml b/helm-prereqs/values/machine-a-tron-multipod.yaml index 723d24e3a4..c14a9945ee 100644 --- a/helm-prereqs/values/machine-a-tron-multipod.yaml +++ b/helm-prereqs/values/machine-a-tron-multipod.yaml @@ -1,4 +1,4 @@ -# machine-a-tron site values — MULTI-POD bmcServices mode (validated example) +# machine-a-tron site values — MULTI-POD bmcServices mode # # Two machine-a-tron pods, each simulating 5 hosts x 2 DPUs (30 BMCs total), # each with a dedicated per-BMC ClusterIP Service range. Copy this file and @@ -11,47 +11,24 @@ # # * Kubernetes 1.29+ for the ServiceCIDR object (beta 1.31+). On OLDER # clusters set bmcServices.serviceCIDR.create=false and pick pod CIDRs -# INSIDE the apiserver's --service-cluster-ip-range — any clusterIP outside -# that range is rejected outright ("the provided IP is not in the valid -# range"). Verify the chosen sub-ranges are free of existing ClusterIPs: -# kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' +# INSIDE the apiserver's --service-cluster-ip-range. # # * The machine-a-tron image must include the bmc-mock Host-header routing -# fallback (PR #3190). Without a bmc_proxy the Redfish client sends no -# "Forwarded" header, and older mocks 404 every request with -# "no router configured". Do NOT set site_explorer.bmc_proxy in this mode — +# fallback (PR #3190). Do NOT set site_explorer.bmc_proxy in this mode — # NICo dials each BMC's ClusterIP directly. # -# * Every pod MUST get disjoint MAC pools (the per-pod TOML overrides below). -# All machine-a-tron instances otherwise derive identical MACs from the -# default pools (02:00:00:00:00:00 / 06:00:00:00:00:00); the pod that -# leases second is rejected on every DHCP with "Network segment mismatch -# for existing MAC address" and simulates nothing. The chart has no -# per-pod MAC knob yet, so the configFiles.matConfigs overrides are the -# only way to set mac_address_pool / hw_mac_address_ranges per pod. +# * MAC address pools are AUTO-GENERATED per pod by the Helm chart: +# - mac_address_pool: 02:00:PP:00:00:00 (PP = pod index) +# - hw_mac_address_ranges: 02:01:PP:00:00:00 (PP = pod index) +# No manual TOML overrides needed. # # * One NICo network segment per pod CIDR: network_prefixes has a -# one-IPv4-prefix-per-segment constraint, so the pod CIDRs cannot be added -# as two prefixes of one OOB segment. Clone one underlay segment per pod -# (same jsonb_populate_record technique as setup-machine-a-tron.sh scale -# mode), prefix = the pod CIDR, gateway = .1, num_reserved = 1 — the chart -# allocates BMC Service IPs starting at .2, which then lines up with the -# DHCP allocator. +# one-IPv4-prefix-per-segment constraint, so each pod CIDR needs its own +# cloned underlay segment with gateway = .1, num_reserved = 1. # -# * Vendor-specific factory credential path: hw types whose Redfish vendor -# string is not recognized by libredfish (e.g. wiwynn_gb200_nvl reports -# "WIWYNN") resolve to BMCVendor::Unknown, so the host factory credential -# must be seeded at machines/all_hosts/factory_default/bmc-metadata-items/unknown. -# -# * GB-class hosts (NvidiaGBx00 flow, e.g. wiwynn_gb200_nvl) require -# expected_machines rows with a WORKING BMC credential before exploration -# completes (error NICO-SITEEXPLORER-141 "Missing credential -# expected_machine"). machine-a-tron registers them itself when -# registerExpectedMachines=true, but only on nico-api builds with the -# Machineatron AddExpectedMachine RBAC grant — on older builds the call is -# 403'd and the rows must be inserted directly (see the fallback in -# setup-machine-a-tron.sh; the row password must be the PINNED password -# below, not the factory default). DPU BMCs explore without expected rows. +# * GB-class hosts (NvidiaGBx00 flow) require expected_machines rows with a +# WORKING BMC credential before exploration completes. machine-a-tron +# registers them when registerExpectedMachines=true. certificate: uris: @@ -65,9 +42,6 @@ global: imagePullSecrets: - name: machine-a-tron-pull -# Build and push the image to the registry of your choice (see the deployment -# guide); the setup script injects these from NICO_IMAGE_REGISTRY/MAT_IMAGE_TAG -# when deploying via setup-machine-a-tron.sh. image: pullPolicy: IfNotPresent repository: "" @@ -86,12 +60,15 @@ bmcServices: servicePort: 443 serviceCIDR: # Requires Kubernetes 1.29+. On older clusters leave false and carve the - # pod CIDRs from the cluster's existing service range (see header). + # pod CIDRs from the cluster's existing service range. create: false pods: mat-0: cidr: "10.233.58.0/23" + # Auto-generated MAC pools: + # mac_address_pool: 02:00:00:00:00:00 + # hw_mac_address_ranges: 02:01:00:00:00:00 machines: compute: hwType: wiwynn_gb200_nvl @@ -101,6 +78,9 @@ pods: adminDhcpRelayAddress: "10.102.0.1" mat-1: cidr: "10.233.60.0/23" + # Auto-generated MAC pools: + # mac_address_pool: 02:00:01:00:00:00 + # hw_mac_address_ranges: 02:01:01:00:00:00 machines: compute: hwType: wiwynn_gb200_nvl @@ -117,78 +97,6 @@ resources: cpu: 500m memory: 512Mi -# Full per-pod TOML overrides. The chart uses these VERBATIM instead of -# generating mat.toml from pods..machines, so they must repeat the -# machine sections — keep them in sync with the pods block above. They exist -# solely to give each pod disjoint MAC pools (see header). +# NOTE: Only use configFiles.matConfigs for advanced TOML overrides not exposed in values. configFiles: - matConfigs: - mat-0: | - # Pod: mat-0 CIDR: 10.233.58.0/23 - carbide_api_url = "https://nico-api.nico-system.svc.cluster.local:1079" - interface = "NOTUSED" - tui_enabled = false - bmc_mock_port = 1266 - use_single_bmc_mock = true - mock_bmc_ssh_server = true - mock_bmc_ssh_port = 2222 - cleanup_on_quit = false - register_expected_machines = true - host_bmc_password = "NicoSiteRoot1" - dpu_bmc_password = "NicoSiteRoot1" - api_refresh_interval = "2s" - - mac_address_pool = { base = "02:00:00:00:00:00", host_bits = 24 } - hw_mac_address_ranges = { base = "06:00:00:00:00:00", host_bits = 32, range_host_bits = 8 } - - [machines.compute] - hw_type = "wiwynn_gb200_nvl" - host_count = 5 - vpc_count = 0 - subnets_per_vpc = 0 - dpu_per_host_count = 2 - dpu_reboot_delay = 1 - host_reboot_delay = 1 - scout_run_interval = "60s" - run_interval_working = "1s" - run_interval_idle = "10s" - network_status_run_interval = "20s" - template_dir = "/opt/machine-a-tron/templates" - oob_dhcp_relay_address = "10.233.58.1" - admin_dhcp_relay_address = "10.102.0.1" - dpus_in_nic_mode = false - mat-1: | - # Pod: mat-1 CIDR: 10.233.60.0/23 - carbide_api_url = "https://nico-api.nico-system.svc.cluster.local:1079" - interface = "NOTUSED" - tui_enabled = false - bmc_mock_port = 1266 - use_single_bmc_mock = true - mock_bmc_ssh_server = true - mock_bmc_ssh_port = 2222 - cleanup_on_quit = false - register_expected_machines = true - host_bmc_password = "NicoSiteRoot1" - dpu_bmc_password = "NicoSiteRoot1" - api_refresh_interval = "2s" - - # Disjoint MAC spaces from mat-0 (outside its 2^24 / 2^32 host spans) - mac_address_pool = { base = "02:00:01:00:00:00", host_bits = 24 } - hw_mac_address_ranges = { base = "06:01:00:00:00:00", host_bits = 32, range_host_bits = 8 } - - [machines.compute] - hw_type = "wiwynn_gb200_nvl" - host_count = 5 - vpc_count = 0 - subnets_per_vpc = 0 - dpu_per_host_count = 2 - dpu_reboot_delay = 1 - host_reboot_delay = 1 - scout_run_interval = "60s" - run_interval_working = "1s" - run_interval_idle = "10s" - network_status_run_interval = "20s" - template_dir = "/opt/machine-a-tron/templates" - oob_dhcp_relay_address = "10.233.60.1" - admin_dhcp_relay_address = "10.102.0.1" - dpus_in_nic_mode = false + matConfigs: {} diff --git a/helm/charts/nico-machine-a-tron/README.md b/helm/charts/nico-machine-a-tron/README.md index 5fb177e224..0aede85e5f 100644 --- a/helm/charts/nico-machine-a-tron/README.md +++ b/helm/charts/nico-machine-a-tron/README.md @@ -4,7 +4,8 @@ Helm chart for deploying Machine-A-Tron - a mock machine simulator for NICo test ## Overview -Machine-A-Tron creates simulated bare-metal machines that behave like real hosts, allowing you to: +Machine-A-Tron creates simulated bare-metal machines that behave like real hosts, +allowing you to: - Test NICo without physical hardware - Simulate multiple hosts, DPUs, switches and power shelves @@ -42,13 +43,15 @@ helm upgrade --install nico ./helm \ [site_explorer] override_target_host = "nico-machine-a-tron-bmc-mock" override_target_port = 1266 -``` + +```text --- ## Mode 2: ClusterIP Mode (Scale Testing) -**Use for load testing environments where simulated machines run alongside real hardware.** +**Use for load testing environments where simulated machines run alongside real +hardware.** Each simulated BMC gets a dedicated ClusterIP service. Supports multi-pod deployments. @@ -167,6 +170,7 @@ pods: ### What Gets Created **Per pod:** + | Resource | Name Pattern | |----------|--------------| | Deployment | `nico-machine-a-tron-pod-0` | @@ -175,11 +179,13 @@ pods: | Service | `nico-machine-a-tron-pod-0-bmc-mock` | **Per BMC:** + | Resource | Name Pattern | |----------|--------------| | ClusterIP Service | `nico-machine-a-tron-bmc-10-100-0-2` | **Cluster-scoped:** + | Resource | Name | |----------|------| | ServiceCIDR | `nico-machine-a-tron-bmc-cidr` | @@ -187,6 +193,7 @@ pods: ### Scale Guidelines **BMC count per GB200 NVL72 rack:** + | Component | Count | BMCs per Unit | Total | |-----------|-------|---------------|-------| | Compute trays | 18 | 3 (1 tray + 2 BF3) | 54 | @@ -195,6 +202,7 @@ pods: | **Total per rack** | | | **71** | **CIDR sizing:** + | CIDR | Usable IPs | Racks per Pod | |------|------------|---------------| | /24 | 253 | ~3 | @@ -224,7 +232,8 @@ gateway = "10.100.0.1" type = "underlay" prefix = "10.100.4.0/22" gateway = "10.100.4.1" -``` + +```text --- @@ -236,6 +245,9 @@ gateway = "10.100.4.1" pods: : cidr: "" # Required for bmcServices mode + macAddressPool: # Optional: override auto-generated MAC pool + base: "02:00:00:00:00:00" + hostBits: 16 machines: : hwType: wiwynn_gb200_nvl @@ -246,6 +258,55 @@ pods: # ... other machine settings ``` +### MAC Address Pool Configuration + +Each pod needs unique MAC addresses to avoid collisions in multi-pod deployments. +By default, the chart auto-generates unique MAC pools per pod based on pod index. + +**Two pools are configured:** + +1. **`mac_address_pool`** - Standalone MACs for BMC and management interfaces: + - Host BMC MAC + - DPU BMC MAC (per DPU) + - DPU Host MAC (representor interface, per DPU) + - DPU OOB MAC (out-of-band management, per DPU) + - NVOS MACs (for switches) + + *Example: GB200 host with 2 DPUs needs ~7 MACs from this pool.* + +2. **`hw_mac_address_ranges`** - Contiguous MAC blocks for NICs: + - ConnectX NICs (e.g., CX-8 needs 10 consecutive MACs) + - Storage NICs + - Management NICs + + *Real hardware has contiguous MAC blocks assigned at manufacturing.* + +```yaml +macAddressPool: + enabled: true # Enable auto-generation + basePrefix: "02:00" # First 2 bytes (locally administered) + hostBits: 16 # ~65K MACs per pod + +hwMacAddressRanges: + enabled: true # Enable auto-generation + basePrefix: "02:01" # First 2 bytes + hostBits: 24 # Total range size + rangeHostBits: 8 # 256 MACs per allocation +``` + +**Auto-generated formats:** + +- `mac_address_pool`: `02:00:PP:XX:XX:XX` (PP = pod index) +- `hw_mac_address_ranges`: `02:01:PP:XX:XX:XX` (PP = pod index) + +**Example with 3 pods:** + +| Pod | mac_address_pool | hw_mac_address_ranges | +|-----|------------------|----------------------| +| mat-0 | `02:00:00:00:00:00` | `02:01:00:00:00:00` | +| mat-1 | `02:00:01:00:00:00` | `02:01:01:00:00:00` | +| mat-2 | `02:00:02:00:00:00` | `02:01:02:00:00:00` | + ### BMC Services Configuration ```yaml @@ -303,12 +364,15 @@ Causes: Verify selector labels match: -# Check service selector +## Check service selector + kubectl -n nico-mat get svc nico-machine-a-tron-bmc-10-100-0-2 -o jsonpath='{.spec.selector}' -# Check pod labels +## Check pod labels + kubectl -n nico-mat get pods -l nvidia-infra-controller/pod-name=pod-0 --show-labels -``` + +```text ### View Generated Config diff --git a/helm/charts/nico-machine-a-tron/templates/configmap.yaml b/helm/charts/nico-machine-a-tron/templates/configmap.yaml index 79e4f48ae8..107c7302d4 100644 --- a/helm/charts/nico-machine-a-tron/templates/configmap.yaml +++ b/helm/charts/nico-machine-a-tron/templates/configmap.yaml @@ -3,23 +3,39 @@ {{- $baseName := include "nico-machine-a-tron.name" . }} {{/* -Count active pods (same logic as deployment.yaml) +Count active pods and build ordered list for consistent pod indexing */}} {{- $activePods := 0 }} +{{- $podList := list }} {{- range $podName, $podConfig := .Values.pods }} {{- if $root.Values.bmcServices.enabled }} {{- if $podConfig.cidr }} {{- $activePods = add $activePods 1 }} +{{- $podList = append $podList $podName }} {{- end }} {{- else }} {{- if $podConfig.machines }} {{- $activePods = add $activePods 1 }} +{{- $podList = append $podList $podName }} {{- end }} {{- end }} {{- end }} +{{- $podList = $podList | sortAlpha }} {{/* Machine defaults */}} {{- $defaults := .Values.machineDefaults | default dict }} +{{/* MAC address pool config */}} +{{- $macPool := .Values.macAddressPool | default dict }} +{{- $macPoolEnabled := $macPool.enabled | default true }} +{{- $macBasePrefix := $macPool.basePrefix | default "02:00" }} +{{- $macHostBits := $macPool.hostBits | default 16 }} + +{{/* HW MAC address ranges config */}} +{{- $hwMacRanges := .Values.hwMacAddressRanges | default dict }} +{{- $hwMacRangesEnabled := $hwMacRanges.enabled | default true }} +{{- $hwMacBasePrefix := $hwMacRanges.basePrefix | default "06" }} +{{- $hwMacHostBits := $hwMacRanges.hostBits | default 32 }} +{{- $hwMacRangeHostBits := $hwMacRanges.rangeHostBits | default 8 }} {{- range $podName, $podConfig := .Values.pods }} {{/* @@ -35,6 +51,14 @@ Skip conditions (same as deployment.yaml) {{- end }} {{- end }} +{{/* Get pod index for MAC address pool */}} +{{- $podIndex := 0 }} +{{- range $i, $p := $podList }} +{{- if eq $p $podName }} +{{- $podIndex = $i }} +{{- end }} +{{- end }} + {{- $name := $baseName }} {{- if gt $activePods 1 }} {{- $name = printf "%s-%s" $baseName $podName }} @@ -88,6 +112,54 @@ data: {{- end }} api_refresh_interval = {{ $root.Values.machineATron.apiRefreshInterval | quote }} + {{/* + MAC Address Pool Configuration + Each pod gets a unique base address: basePrefix:podIndex:00:00:00 + Example with basePrefix=02:00 and 3 pods: + Pod 0: 02:00:00:00:00:00 + Pod 1: 02:00:01:00:00:00 + Pod 2: 02:00:02:00:00:00 + */}} + {{- if $macPoolEnabled }} + {{- $podMacPool := $podConfig.macAddressPool | default dict }} + {{- if $podMacPool.base }} + {{/* Explicit per-pod override */}} + [mac_address_pool] + base = {{ $podMacPool.base | quote }} + host_bits = {{ $podMacPool.hostBits | default $macHostBits }} + {{- else }} + {{/* Auto-generate based on pod index */}} + [mac_address_pool] + base = "{{ $macBasePrefix }}:{{ printf "%02x" $podIndex }}:00:00:00" + host_bits = {{ $macHostBits }} + {{- end }} + {{- end }} + + {{/* + HW MAC Address Ranges - for network interface MAC blocks + Format: basePrefix:podIndex:00:00:00 + Example with 3 pods: + Pod 0: 02:01:00:00:00:00 + Pod 1: 02:01:01:00:00:00 + Pod 2: 02:01:02:00:00:00 + */}} + {{- if $hwMacRangesEnabled }} + {{- $podHwMacRanges := $podConfig.hwMacAddressRanges | default dict }} + {{- if $podHwMacRanges.base }} + {{/* Explicit per-pod override */}} + [hw_mac_address_ranges] + base = {{ $podHwMacRanges.base | quote }} + host_bits = {{ $podHwMacRanges.hostBits | default $hwMacHostBits }} + range_host_bits = {{ $podHwMacRanges.rangeHostBits | default $hwMacRangeHostBits }} + {{- else }} + {{/* Auto-generate based on pod index: 02:01:PP:00:00:00 */}} + [hw_mac_address_ranges] + base = "{{ $hwMacBasePrefix }}:{{ printf "%02x" $podIndex }}:00:00:00" + host_bits = {{ $hwMacHostBits }} + range_host_bits = {{ $hwMacRangeHostBits }} + {{- end }} + {{- end }} + {{- range $sectionName, $section := $podConfig.machines }} {{- if $section }} diff --git a/helm/charts/nico-machine-a-tron/values.yaml b/helm/charts/nico-machine-a-tron/values.yaml index 5d3fa1f308..82bfc3cea9 100644 --- a/helm/charts/nico-machine-a-tron/values.yaml +++ b/helm/charts/nico-machine-a-tron/values.yaml @@ -162,13 +162,61 @@ machineDefaults: dpusInNicMode: false ## ============================================================================= -## Pod Configurations - One pod per config entry +## MAC Address Pool - Standalone MACs for BMC and management interfaces ## ============================================================================= -## Each entry creates: -## - 1 Deployment (pod) -## - 1 ConfigMap with mat.toml -## - 1 Certificate -## - 1 internal Service +## Allocates individual MACs for simulated machine interfaces: +## - Host BMC MAC (bmc_mac_address) +## - DPU BMC MAC (per DPU) +## - DPU Host MAC (representor interface, per DPU) +## - DPU OOB MAC (out-of-band management, per DPU) +## - NVOS MACs (for switches) +## +## Example: GB200 host with 2 DPUs needs ~7 MACs from this pool. +## +## Each pod gets a unique base to avoid collisions in multi-pod deployments. +## +## Format: 02:00:PP:XX:XX:XX +## - 02:00 = Locally administered prefix (required for virtual MACs) +## - PP = Pod index (0x00, 0x01, 0x02, ...) +## - XX = Host bits (allocated sequentially) +## +## Scale with hostBits=16: +## - 65K MACs per pod +## - Supports up to 256 pods +## +macAddressPool: + ## Enable auto-generation of per-pod MAC address pools + enabled: true + ## Base prefix (first 2 bytes). Pod index added to byte 3. + basePrefix: "02:00" + ## Number of host bits per pod (16 = ~65K MACs per pod) + hostBits: 16 + +## ============================================================================= +## HW MAC Address Ranges - Contiguous MAC blocks for network interface cards +## ============================================================================= +## Allocates contiguous MAC address ranges for NICs that require sequential MACs: +## - ConnectX NICs (e.g., CX-8 needs 10 consecutive MACs) +## - Storage NICs (multiple MACs for storage interfaces) +## - Management NICs +## +## Real hardware has contiguous MAC blocks assigned at manufacturing. +## This pool simulates that by allocating 256-MAC blocks (rangeHostBits=8). +## +## Each pod gets a unique range base derived from pod index. +## +## Format: 02:01:PP:XX:XX:XX where PP = pod index +## +hwMacAddressRanges: + ## Enable auto-generation of per-pod HW MAC ranges + enabled: true + ## Base prefix (first 2 bytes). Pod index added to byte 3. + basePrefix: "02:01" + ## Total host bits (24 = 16M addresses in the range pool) + hostBits: 24 + ## Bits per allocated range (8 = 256 MACs per range allocation) + rangeHostBits: 8 + ## - N ClusterIP BMC services (when bmcServices.enabled) ## ## SCALE: Use /22 CIDR per pod = 1021 usable IPs = ~14 racks worth of BMCs