Skip to content

[Bugfix] Environment pre-check code adaptation #1006

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Menglths merged 2 commits into from
Jun 29, 2026
Merged

[Bugfix] Environment pre-check code adaptation #1006

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5b0924f
Environment pre-check code adaptation
Menglths Jun 8, 2026
777cb91
Merge branch 'develop' into EnvPreCheck_2
Menglths Jun 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 68 additions & 83 deletions test/common/envPreCheck/EnvPreCheck.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,52 +1,70 @@
# 🔍 Environment PreCheck Automation Test Suite | Environment PreCheck Suite
---

# 🔍 Environment PreCheck Suite

> 💡 **Core Objective**: Prior to model deployment or training task execution, conduct comprehensive health checks on critical cluster capabilities to proactively identify potential issues in SSH configuration, device status, network connectivity, TLS encryption, model integrity, and storage performance.
> 💡 **Core Objective**: Before model deployment or training tasks begin, perform comprehensive health checks on key cluster capabilities to proactively identify potential issues in SSH configuration, device status, network connectivity, TLS encryption, model integrity, and storage performance.

---

## 📋 Table of Contents

- [🌟 Core Features](#-core-features)
- [🎯 Functional Overview](#-functional-overview)
- [🚀 Quick Start](#-quick-start)
- [📁 Project Structure](#-project-structure)
- [🧪 Test Case Details](#-test-case-details)
- [⚙️ Configuration](#️-configuration)
* [🌟 Key Features](#-key-features)
* [🎯 Overview](#-overview)
* [⚙️ Configuration](#️-configuration)
* [🚀 Quick Start](#-quick-start)
* [🧪 Test Case Details](#-test-case-details)

---

## 🌟 Core Features
## 🌟 Key Features

| Feature | Description |
|------|------|
| 🎯 **Intelligent Platform Detection** | Automatically detect NPU (Ascend) / GPU environments and execute corresponding test logic |
| 🔧 **Modular Architecture** | Support flexible combination of test cases by stage, platform, and feature dimensions |
| 🛡️ **Fail-Fast Mechanism** | Immediately terminate upon critical check failures to avoid invalid resource consumption |
| 📊 **Comprehensive Coverage** | 6 major dimensions, 12+ detailed checks to ensure environmental consistency |
| Feature | Description |
| ------------------------------------- | ------------------------------------------------------------------------------------------- |
| 🎯 **Intelligent Platform Detection** | Automatically detects NPU (Ascend) / GPU environments and executes corresponding test logic |
| 🔧 **Modular Architecture** | Supports flexible combination of test cases by stage, platform, and feature |
| 🛡️ **Failure Fast Mechanism** | Critical check failures immediately terminate execution to avoid wasted resources |
| 📊 **Comprehensive Coverage** | Covers 6 major dimensions and 12+ checks to ensure environment consistency |

---

## 🎯 Functional Overview
## 🎯 Overview

This test suite automatically executes the following health checks prior to task initiation:
This test suite automatically performs the following health checks before task execution:

### 🔐 Infrastructure Layer

- **SSH Passwordless Login Verification**: Ensure bidirectional passwordless access between Master ↔ Worker is functional
- **Device Health Inspection**: NPU/GPU online status, driver version, and temperature monitoring
* **SSH Passwordless Login Verification**: Ensures bidirectional password-free access between Master ↔ Worker
* **Device Health Inspection**: NPU/GPU status, driver versions, temperature monitoring

### 🌐 Network Communication Layer

- **Inter-Node Connectivity Testing**: HCCN (NPU) / NVLink & InfiniBand (GPU) link packet loss detection
- **TLS Encryption Status**: Verify Ascend device TLS switch configuration complies with security baselines
* **Node Connectivity Testing**: Packet loss detection for HCCN (NPU) / NVLink & InfiniBand (GPU)
* **TLS Encryption Status**: Verifies whether TLS settings between Ascend devices meet security baselines

### 💾 Data Integrity Layer

- **Model Weight Vault**: File list integrity scan → MD5/SHA256 hash verification → weight format validity validation
* **Model Weights Vault**: File list integrity scan → MD5/SHA256 hash verification → weight format validation

### ⚡ Performance Baseline Layer

- **Storage Bandwidth Stress Test**: Compare measured Embedding/Fetch operation bandwidth against expected thresholds (< 85% triggers warning)
* **Storage Bandwidth Benchmarking**: Compare measured Dump/Load bandwidth against expected thresholds (< 85% triggers warning)

---

## ⚙️ Configuration (`config.yaml`)

| Parameter | Type | Description | Example |
| --------------------------- | ------ | ---------------------------------------------- | ---------------------------------- |
| `master_ip` | string | Master node SSH IP | `192.168.1.10` |
| `worker_ip` | list | Worker node IP list (empty = single-node test) | `["192.168.1.11", "192.168.1.12"]` |
| `ascend_rt_visible_devices` | string | Visible NPU device indices | `"0,1,2,3,4,5,6,7"` |
| `node_num` | int | Total number of nodes (master + workers) | `2` |
| `model_path` | string | Root directory of model weights | `/data/models/llama-7b` |
| `hf_model_name` | string | HuggingFace model identifier | `meta-llama/Llama-2-7b` |
| `middle_page` | string | Model intermediate page / organization name | `model_storage` |
| `expected_embed_bandwidth` | float | Expected Dump bandwidth (GB/s) | `12.0` |
| `expected_fetch_bandwidth` | float | Expected Load bandwidth (GB/s) | `8.0` |
| `storage_backends` | list | Mounted storage paths in current environment | `["/mnt/nfs"]` |

---

Expand All @@ -57,31 +75,30 @@ This test suite automatically executes the following health checks prior to task
```bash
tests/
├── common/envPreCheck/
│ ├── run_env_preCheck.py # Core detection engine
│ └── utils/ # Auxiliary utilities
│ ├── run_env_preCheck.py # Core check engine
├── suites/E2E/
│ └── test_environment_precheck.py # Test entry point
└── config.yaml # PreCheck threshold configuration file
└── config.yaml # Precheck threshold configuration
```

### 🎮 Execution Methods
### 🎮 How to Run

```bash
# Enter test directory
# Navigate to test directory
cd tests/

# 1️⃣ Execute full precheck (stage 2)
pytest --stage=2

# 2️⃣ Execute by hardware platform
# 1️⃣ Run by hardware platform
pytest --platform=npu # Ascend NPU environment
pytest --platform=gpu # NVIDIA GPU environment

# 3️⃣ Execute by individual feature
# 2️⃣ Run by specific feature
pytest --feature=test_ssh_login
pytest --feature=test_check_bandwidth

# 4️⃣ Run specific file directly
# 3️⃣ Run full precheck (stage 2)
pytest --stage=2

# 4️⃣ Run a specific test file directly
pytest suites/E2E/test_environment_precheck.py -v
```

Expand All @@ -95,8 +112,8 @@ pytest suites/E2E/test_environment_precheck.py -v
test_ssh_login()
```

- **Verification Content**: Master → Worker bidirectional passwordless login
- **Failure Strategy**: ❌ **Immediate termination** of all subsequent tests (blocking issue)
* **Validation**: Bidirectional passwordless login between Master → Worker
* **Failure Strategy**: ❌ **Immediately aborts** all subsequent tests (blocking issue)

### 🖥️ Device Status Check

Expand All @@ -108,21 +125,21 @@ test_hccn_check_device_status()
test_nvidia_check_device_status()
```

- **Check Items**: Device online status, driver loading, memory health, temperature thresholds
* **Checks**: Device availability, driver status, memory health, temperature thresholds

### 🌐 Inter-Node Network Quality

```python
test_check_hccn_ping() # NPU: HCCN links
test_check_nvidia_ping() # GPU: NCCL networks
test_check_hccn_ping() # NPU: HCCN link
test_check_nvidia_ping() # GPU: NCCL network
```

**Generate full-link topology report**:
**Generates full topology report**:

```
✅ local_card_0 → local_card_1 [0.02ms, 0% loss]
✅ local_card_0 → remote_ip:192.168.1.10 [0.15ms, 0% loss]
⚠️ remote_card_1 → local_ip:192.168.1.5 [2.34ms, 3% loss] ← Abnormal link
⚠️ remote_card_1 → local_ip:192.168.1.5 [2.34ms, 3% loss] ← abnormal link
```

### 🔒 TLS Security Configuration
Expand All @@ -131,8 +148,8 @@ test_check_nvidia_ping() # GPU: NCCL networks
test_check_tls()
```

- **Check Target**: `tls_switch` status of each card
- **Pass Criteria**: All device TLS switches are consistent and comply with security policies (usually 0 or 1)
* **Check Target**: `tls_switch` status on each device
* **Pass Criteria**: All devices have consistent TLS settings and comply with security policy (typically 0 or 1)

### 📦 Model Weight Integrity

Expand All @@ -142,54 +159,22 @@ test_check_model_weights()

**Three-layer protection system**:

1. **File Tree Scanning**: Confirm existence of all `.bin`, `.safetensors`, `.json` files
2. **Hash Verification**: Compare against pre-computed checksums to prevent transmission corruption
3. **Format Validity**: Rapid loading validation using `torch.load` / `safetensors`
1. **File Tree Scan**: Ensure all `.bin`, `.safetensors`, `.json` files exist
2. **Hash Verification**: Compare precomputed checksums to prevent corruption
3. **Format Validation**: Quick load validation using `torch.load` / `safetensors`

### ⚡ Storage Bandwidth Benchmark Test
### ⚡ Storage Bandwidth Benchmark

```python
test_check_bandwidth()
```

- **Test Scenario**: Large-scale Embedding reads / Checkpoint Fetch writes
- **Decision Logic**:
* **Test Scenario**: Large-scale embedding read / checkpoint fetch write
* **Evaluation Logic**:

```python
if actual_bandwidth < expected_threshold * 0.85:
raise PerformanceWarning("Insufficient storage bandwidth may impact training efficiency")
raise PerformanceWarning("Insufficient storage bandwidth, may impact training efficiency")
```

---

## ⚙️ Configuration (`config.yaml`)

| Configuration Item | Type | Description | Example Value |
|--------|------|------|--------|
| `master_ip` | string | SSH login IP of the cluster master node | `192.168.1.10` |
| `worker_ip` | list | IP addresses of cluster worker nodes | `["192.168.1.11", "192.168.1.12"]` |
| `ascend_rt_visible_devices` | string | Visible device IDs for Ascend/NPU | `"0,1,2,3,4,5,6,7"` |
| `node_num` | int | Total number of cluster nodes | `2` |
| `model_path` | string | Root directory of model weights | `/data/models/llama-7b` |
| `hf_model_name` | string | HuggingFace model identifier | `meta-llama/Llama-2-7b` |
| `middle_page` | string | Intermediate page/organization name corresponding to the model | `model_storage` |
| `expected_embed_bandwidth` | float | Expected embedding bandwidth (GB/s) | `12.0` |
| `expected_fetch_bandwidth` | float | Expected fetch bandwidth (GB/s) | `8.0` |
| `kvCache_block_number` | int | Number of KV Cache pre-allocated blocks | `4096` |
| `storage_backends` | list | Storage backend mount paths | `["/data", "/mnt/nfs"]` |

---

## 🎨 Output Example

```diff
🚀 Launching Environment PreCheck Suite (Platform: NPU)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ [PASS] SSH Passwordless Login (2/2 nodes)
✅ [PASS] NPU Device Status (8/8 cards online)
✅ [PASS] HCCN Link Connectivity (56/56 links)
⚠️ [WARN] TLS Configuration (card_3: tls_switch=1, expected=0)
✅ [PASS] Model Weight Integrity (hash verified)
❌ [FAIL] Storage Bandwidth Check (6.5 GB/s < 12.0 GB/s * 0.85)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔴 PreCheck failed, please fix high-priority issues before launching training tasks
```
68 changes: 23 additions & 45 deletions test/common/envPreCheck/EnvPreCheck_ch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
我来帮你重新润色这份技术文档，让它更加专业且富有视觉层次：

---

Expand All @@ -13,10 +12,9 @@

- [🌟 核心特性](#-核心特性)
- [🎯 功能概述](#-功能概述)
- [⚙️ 配置说明](#️-配置说明)
- [🚀 快速开始](#-快速开始)
- [📁 项目结构](#-项目结构)
- [🧪 测试用例详解](#-测试用例详解)
- [⚙️ 配置说明](#️-配置说明)

---

Expand Down Expand Up @@ -51,7 +49,23 @@

### ⚡ 性能基线层

- **存储带宽压测**：Embedding/Fetch 操作实测带宽对比预期阈值（< 85% 触发告警）
- **存储带宽压测**：Dump/Load 操作实测带宽对比预期阈值（< 85% 触发告警）

---
## ⚙️ 配置说明（`config.yaml`）

| 配置项 | 类型 | 说明 | 示例值 |
|--------|------|------|--------|
| `master_ip` | string | Master 节点 SSH IP | `192.168.1.10` |
| `worker_ip` | list | Worker 节点 IP 列表（若不填则表示单节点自测） | `["192.168.1.11", "192.168.1.12"]` |
| `ascend_rt_visible_devices` | string | NPU 可见设备序号 | `"0,1,2,3,4,5,6,7"` |
| `node_num` | int | master和worker总节点数 | `2` |
| `model_path` | string | 模型权重根目录 | `/data/models/llama-7b` |
| `hf_model_name` | string | HuggingFace 模型标识 | `meta-llama/Llama-2-7b` |
| `middle_page` | string | 模型中间页/组织名称 | `model_storage` |
| `expected_embed_bandwidth` | float | 预期 Dump 带宽 (GB/s) | `12.0` |
| `expected_fetch_bandwidth` | float | 预期 Load 带宽 (GB/s) | `8.0` |
| `storage_backends` | list | 当前容器所在环境的挂载点路径 | `["/mnt/nfs"]` |

---

Expand All @@ -63,7 +77,6 @@
tests/
├── common/envPreCheck/
│ ├── run_env_preCheck.py # 核心检测引擎
│ └── utils/ # 辅助工具集
├── suites/E2E/
│ └── test_environment_precheck.py # 测试入口
└── config.yaml # 预检阈值配置文件
Expand All @@ -75,17 +88,17 @@ tests/
# 进入测试目录
cd tests/

# 1️⃣ 执行完整预检（阶段 2）
pytest --stage=2

# 2️⃣ 按硬件平台执行
# 1️⃣ 按硬件平台执行
pytest --platform=npu # Ascend NPU 环境
pytest --platform=gpu # NVIDIA GPU 环境

# 3️⃣ 按特性单独执行
# 2️⃣ 按特性单独执行
pytest --feature=test_ssh_login
pytest --feature=test_check_bandwidth

# 3️⃣ 执行完整预检（阶段 2）
pytest --stage=2

# 4️⃣ 直接运行特定文件
pytest suites/E2E/test_environment_precheck.py -v
```
Expand Down Expand Up @@ -164,39 +177,4 @@ test_check_bandwidth()
raise PerformanceWarning("存储带宽不足，可能影响训练效率")
```

---

## ⚙️ 配置说明（`config.yaml`）

| 配置项 | 类型 | 说明 | 示例值 |
|--------|------|------|--------|
| `master_ip` | string | Master 节点 SSH IP | `192.168.1.10` |
| `worker_ip` | list | Worker 节点 IP 列表 | `["192.168.1.11", "192.168.1.12"]` |
| `ascend_rt_visible_devices` | string | NPU 可见设备序号 | `"0,1,2,3,4,5,6,7"` |
| `node_num` | int | 集群总节点数 | `2` |
| `model_path` | string | 模型权重根目录 | `/data/models/llama-7b` |
| `hf_model_name` | string | HuggingFace 模型标识 | `meta-llama/Llama-2-7b` |
| `middle_page` | string | 中间页/组织名称 | `model_storage` |
| `expected_embed_bandwidth` | float | 预期 Embedding 带宽 (GB/s) | `12.0` |
| `expected_fetch_bandwidth` | float | 预期 Fetch 带宽 (GB/s) | `8.0` |
| `kvCache_block_number` | int | KV Cache 预分配块数 | `4096` |
| `storage_backends` | list | 存储后端挂载路径 | `["/data", "/mnt/nfs"]` |

---

## 🎨 输出示例

```diff
🚀 启动环境预检套件 (Platform: NPU)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ [PASS] SSH 免密登录 (2/2 nodes)
✅ [PASS] NPU 设备状态 (8/8 cards online)
✅ [PASS] HCCN 链路连通性 (56/56 links)
⚠️ [WARN] TLS 配置 (card_3: tls_switch=1, expected=0)
✅ [PASS] 模型权重完整性 (hash verified)
❌ [FAIL] 存储带宽检测 (6.5 GB/s < 12.0 GB/s * 0.85)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔴 预检未通过，请修复高优问题后再启动训练任务
```

---
Loading
Loading