[CICD] Add train image pipeline and real TE-FL smoke testCi/ascend train te smoke by AlexMa616 · Pull Request #1250 · flagos-ai/FlagScale

AlexMa616 · 2026-07-24T14:21:32Z

Description

Add an Ascend training image pipeline and enable the Ascend CI suite with a
real TransformerEngine-FL training smoke test.

The change builds the training runtime from Ascend-specific Docker and installer
files, validates Megatron-LM-FL and TE-FL NPU initialization, and runs a
two-device Qwen3 0.6B smoke test through the TE-FL reference backend.

Chip-specific setup remains outside reusable common workflows. Ascend behavior
is defined through the platform configuration, Ascend installer and launcher,
Dockerfile, and dedicated image-build workflow. No FlagScale core source is
changed.

Prerequisite: TransformerEngine-FL must include 771a7488
(fix(ascend): include NPU backend in Python package) or an equivalent upstream
change before rebuilding the image from the official TE-FL repository.

Type of change

Infra/Build change (changes to CI/CD workflows or build scripts)
Bug fix
Code refactoring
New feature (non-breaking change which adds functionality)
Documentation change
Breaking change

Changes

Added a trusted Ascend training image build and Harbor publishing workflow.
Added the Ascend training Dockerfile and source dependency installation for
Megatron-LM-FL and TransformerEngine-FL.
Added an Ascend launcher that restores TE-FL NPU compatibility after Megatron
platform initialization.
Added a two-device Qwen3 0.6B mock-data smoke test using
transformer_impl: transformer_engine and TE-FL reference.torch operators.
Added platform-neutral finite-loss smoke validation and parser tests.
Enabled Ascend through the standard all_tests workflow and selected the
validated Harbor training image.
Aligned torchvision with the validated torch 2.7.1 and
torch_npu 2.7.1.post2 runtime.
Improved installer cache path handling, resumable sccache downloads, and
relative requirements include processing.

Validation

Repository-built training image smoke:
- Completed 3 training iterations on two Ascend NPUs.
- Dispatched TE-FL GEMM, RMSNorm forward/backward, and optimizer operations
  through reference.torch.
- No skipped or NaN iterations.
- 1 passed
- FLAGSCALE_ASCEND_REPO_IMAGE=PASS
Validated Harbor image:
- harbor.baai.ac.cn/flagos-dev/flagscale:manual-20260724-ascend-train-426ad116-te771a7488
- Digest: sha256:2fe15cf8b339b6828395fc3ce56e67b004b7c5f251d9e6673f20579819fe2c16
GitHub Actions:
- https://github.com/BrianPei/FlagScale/actions/runs/30083843003
- CLI, unit, train, inference, serve, and aggregate jobs passed.
- Benchmark and hetero-train remained skipped because they are not enabled
  in the Ascend matrix.
Local focused checks:
- Shell, Python, YAML, and Git diff validation passed.
- Ascend training matrix parsing passed.
- Result parser tests: 2 passed.

Checklist

I have read and followed the contributing guidelines
The functionality is complete for the Ascend training pipeline and smoke scope
I have commented my code, particularly in CI workflow setup steps
My changes generate no new warnings
I have tested the training workflow on the Ascend platform

CLAassistant · 2026-07-24T14:38:12Z

All committers have signed the CLA.

Darryl233 · 2026-08-05T14:10:05Z

Next time we try to build a build_image_common.yml and use vendor.yml to control the build workflow

We’ll reuse a common image-build workflow with vendor-specific configuration for future chips.

Darryl233 · 2026-08-05T14:11:30Z

  envs:
    ASCEND_VISIBLE_DEVICES: "0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15"
    HCCL_WHITELIST_DISABLE: 1
+    CC: ./tools/install/ascend/triton_gxx_wrapper.sh


what is this？

This wrapper serializes concurrent Triton compilation to prevent .gch file conflicts. I’ve added a comment to clarify it.

Darryl233 · 2026-08-05T14:13:59Z

+
+data:
+  # Keep the accelerator smoke test independent of runner-local .idx/.bin files.
+  # MockGPTDataset still exercises tokenization, forward/backward, HCCL, and the optimizer.


If use the MockGPTDataset，we can‘t verify the accuracy with golden value

I replaced the mock dataset with the existing real dataset and added golden loss validation.

Darryl233 · 2026-08-05T14:17:52Z

We write a test script to test another test script?

The extra parser logic and its unit test have been removed.

hooray03 · 2026-08-10T07:45:13Z

Following the review suggestion, image building has been split from this PR. This PR now contains only the standard Ascend workflow and test validation using prebuilt images; it no longer changes Dockerfiles, image-build workflows, requirements, or Ascend image installers. The image-build work is preserved separately on BrianPei:ci/ascend-image-build and will be submitted after this workflow PR is merged.

Darryl233 · 2026-08-11T07:08:08Z

check if necessary

Darryl233 · 2026-08-11T07:08:47Z

check if necessary

…d-train-te-smoke # Conflicts: # .github/workflows/all_tests.yml # tests/test_utils/runners/run_functional_tests.sh

xmhubj · 2026-08-13T11:19:43Z

+  - data
+
+data:
+  data_path: /home/gitlab-runner/data/pile_wikipedia_demo/pile_wikipedia_demo


is this the right data path?

Confirmed, yes.

xmhubj · 2026-08-13T11:35:08Z

+# after the Ascend inference and serve matrices pass.
+FLAGGEMS_REF="${FLAGSCALE_FLAGGEMS_REF:-61f3ff2773bc9c8e86b97489775ef9668a96a33c}"
+VLLM_PLUGIN_REPO="${FLAGSCALE_VLLM_PLUGIN_REPO:-https://github.com/flagos-ai/vllm-plugin-FL.git}"
+VLLM_PLUGIN_REF="${FLAGSCALE_VLLM_PLUGIN_REF:-43edeb601f4b8f616f56109de64836529e758deb}"


why we use this commit tag

AlexMa616 requested a review from aoyulong as a code owner July 24, 2026 14:21

AlexMa616 force-pushed the ci/ascend-train-te-smoke branch 2 times, most recently from be82b5a to c2a56d4 Compare July 24, 2026 14:44

tengqm force-pushed the main branch from ff2dae3 to 54957df Compare July 25, 2026 11:22

tengqm requested review from ceci3, cyber-pioneer, heavyrain-lzy and zhaoyinglia as code owners July 25, 2026 11:22

AlexMa616 force-pushed the ci/ascend-train-te-smoke branch from c2a56d4 to 3d7ea91 Compare July 26, 2026 14:03

Darryl233 reviewed Aug 5, 2026

View reviewed changes

hooray03 force-pushed the ci/ascend-train-te-smoke branch 2 times, most recently from 3eec7aa to 64843ae Compare August 10, 2026 07:44

hooray03 force-pushed the ci/ascend-train-te-smoke branch 4 times, most recently from 5abe025 to b0de48f Compare August 10, 2026 11:34

Darryl233 reviewed Aug 11, 2026

View reviewed changes

ci(ascend): enable workflow validation

b3072f1

hooray03 force-pushed the ci/ascend-train-te-smoke branch from b0de48f to b3072f1 Compare August 11, 2026 07:56

AlexMa616 added 5 commits August 11, 2026 20:49

ci(ascend): stabilize functional test launch

02e2569

Merge commit '067efe6f1b00bc9416e22cfe46c52990e10bf045' into ci/ascen…

ccb7f8a

…d-train-te-smoke # Conflicts: # .github/workflows/all_tests.yml # tests/test_utils/runners/run_functional_tests.sh

ci(ascend): use validated runtime images

dc0c34c

ci(ascend): restore available train image

fdcf3b7

ci(ascend): require standard training runtime

53fa6be

ci(metax): add image contracts and benchmark gate

34f0ac8

xmhubj reviewed Aug 13, 2026

View reviewed changes

hooray03 added 4 commits August 13, 2026 19:27

fix(metax): pin training runtime in image

58a23ca

fix(ascend): preserve vendor torch runtime

d4e91bd

fix(ascend): preserve vendor torch runtime

3ba5b8d

fix(ascend): validate inference with mounted driver

7c86400

xmhubj reviewed Aug 13, 2026

View reviewed changes

AlexMa616 and others added 23 commits August 13, 2026 19:51

ci(ascend): deduplicate CLI installation

47f4420

fix(ascend): preserve inference base contract

857b11e

ci: scope initial platform image contracts

d81bebc

fix(metax): initialize pinned training runtime

0a6cc4b

ci(metax): pin validated train image

29d322d

docs(ascend): use readable inference image tag

e4f6229

fix(ascend): restore inference installer mode

1143d4c

ci: unify platform image build workflows

06303db

ci: align image names with Harbor convention

d43cc41

ci: finalize unified image build pipeline

de078f4

ci: enable branch image build validation

cdb57cd

ci: remove jq dependency from image runners

32bd898

ci: cancel stale branch image builds

c152d71

ci: test validated images when builds are skipped

d8b9e89

feat(metax): add inference image contract

87abf1c

ci: dispatch image builds manually

5fb8fc5

test(metax): add inference and serve coverage

86c8e54

ci(metax): enable inference and serve tests

d8c6b36

ci: isolate task runtimes in all images

cd36ce1

fix(ascend): defer driver checks to runtime

6b97dad

fix(ascend): keep device probes out of image build

03f5868

fix(ci): use static local rendezvous for image smoke

57d19d0

fix(ascend): isolate HCCL smoke ports

a9c7802

Uh oh!

Conversation

AlexMa616 commented Jul 24, 2026

Description

Type of change

Changes

Validation

Checklist

Uh oh!

CLAassistant commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hooray03 commented Aug 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

CLAassistant commented Jul 24, 2026 •

edited

Loading