You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Cut ADO CI from about 105 to 68 minutes and releases from 141 to about 90 - #1384
The internal Azure DevOps pipelines took about 105 minutes for CI and 141 minutes for a release. A Monday release rehearsal, or the first run of a rel/v* branch, took about 210 minutes, which is why the release timeout had to go up to 240 minutes (#1343, #1347). Almost all of that was one step, Build CLI. It built the CLI and then ran every test suite one after another, on a 2-vCPU agent.
This PR runs the tests in parallel jobs, the same lanes GitHub Actions runs since #1378:
The Build job no longer runs tests. It builds, packages, signs and publishes exactly as before.
New test jobs in the Build stage (.pipelines/templates/test-jobs.yaml): CLI shards 1, 2 and 4, UI Automation, Node/analyzer/Pester, and DevTools native tests. They run build-cli.ps1 -OnlyTests -UseExistingArtifacts, the same as GitHub. Every later stage depends on the whole Build stage, so a failing test still blocks every release stage.
Release only: a new BuildForTests job runs the same stable, unstubbed build, unsigned, next to Build. The tests use its output, so they don't wait for signing. 1ES uploads a job's artifacts only after the whole job ends.
Release only: the Build job still runs the find-ui tests (263 tests, about 3 min) on the corpus it refreshes from GitHub and ships. The test jobs build from a fresh checkout, so they only see the committed corpus.
The test jobs opt out of CodeQL, with a justification. The Build job, which builds what we ship, is still traced. Before, CodeQL traced the test runs too, and timing-sensitive tests failed under it.
Timeouts: CI Build 240 → 120 min, release Build 240 → 180 min, each test job 90 min.
Before / after
Measured on this branch, against main runs of the same code (this PR changes only .pipelines/). Nothing was signed or published: on a feature branch, ESRP signing is off and every publishing stage is skipped.
Before
After
Saving
ADO CI, queue → finish
102.6 min (main run 20261120.6; median 104 over Nov 14–20)
65.6 / 67.7 / 67.9 / 68.7 min (4 of 4 runs green)
−35 min
Release, queue → finish
141.2 min (rel/v0.8.0, 20261106.5)
82.3 / 84.9 min (20261120.5, .3; publishing skipped, a real release adds ~6)
−56 min
Release Build job under CodeQL tracing
Build CLI alone took 149 min, then failed on timing tests (20261117.2)
whole Build job took 95 min and passed (20261120.1)
≥ −55 min
Every suite runs, with the same test counts as main on the same code:
Suite
Before (one job)
After (per job)
WinApp.Cli.Tests
10,384
115 + 9,144 + 50 + 1,075 = 10,384
UI Automation / analyzer / npm
1,085 / 92 / 350
1,085 / 92 / 350
NuGet / scripts Pester
140 / 365
140 / 365
DevTools native (window checks)
14,360
14,360
Not tested / risks
Signing is not exercised. It only runs on a real rel/v* release. The signing steps and their inputs are unchanged, but the first real release is the first signed run of the new layout.
The test jobs opt out of CodeQL, so test projects are no longer in the CodeQL database. Shipped code is still traced. 1ES documents this opt-out per job, with a justification.
About 2× agent-minutes per run on the shared Azure-Pipelines-1ESPT-ExDShared pool: CI ~107 → ~182, release Build stage ~133 → ~263. Each run uses ~7 agents at once instead of 1. Median wait for an agent was 2 min across 79 runs.
The known ADO-only flaky test, DeploymentStateStoreTests.Commit_ConcurrentWritersCannotPublishTheSameRevision, failed in 1 of 7 branch CI runs. That's unchanged from main.
Checklist
Tested on Azure DevOps: 7 CI runs and 4 release runs of this branch, plus both pipelines compiled with the ADO preview API.
Why a second, unsigned build in the release: the test jobs could instead wait for Build's signed artifacts. That makes the test path run after signing and SDL, so a release would take ~105 min instead of ~85.
Run 20261120.1's BuildForTests failed BinSkim BA2007 on the third-party libHarfBuzzSharp.dll. Its baseline in .config/guardian/.gdnbaselines matches files by their path under $(Build.ArtifactStagingDirectory). The job now publishes from there, like build.yaml.
CLI shard 1 (PackageCommandTests + PerfFixtureTests) is now the slowest lane, at ~28 min. Splitting it further is a follow-up.
Shortens Azure DevOps CI and release waits by moving tests out of the shipping build and into parallel jobs, following the repository’s GitHub Actions test layout.
Changes:
Add parallel CLI shards, UI Automation, Auxiliary, and DevTools native test jobs with separate result artifacts.
Add an unsigned release build for testing and disable CodeQL tracing in release test-only jobs.
Skip test suites in the shipping build, reduce job timeouts, and document the new layout.
File
Description
.pipelines/templates/test-jobs.yaml
Define shared test lanes, artifact downloads, and result uploads.
.pipelines/templates/build.yaml
Skip in-job tests and remove centralized test-result publishing.
.pipelines/release.yml
Add an unsigned test build and parallel lanes; reduce the shipping build timeout.
.pipelines/README.md
Document test jobs, artifact sources, and troubleshooting.
.pipelines/ci.yml
Connect parallel test jobs to build artifacts and lower the build timeout.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The failure-triage instructions incorrectly direct all failed lanes to .trx files, although Pester results use XML and npm and native-unit failures appear in job logs.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Failed lanes may report results outside .trx files
.pipelines/README.md:230
Optional — This directs every failed lane to a .trx file, but the Auxiliary lane writes Pester results as nuget-pester.xml and scripts-pester.xml; npm failures and native-unit failures are reported in the job log instead. Please point readers to the logs and the available report formats.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ready-for-reviewAgent work and technical checks complete; awaiting review or re-review, not approval or merge
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The internal Azure DevOps pipelines took about 105 minutes for CI and 141 minutes for a release. A Monday release rehearsal, or the first run of a
rel/v*branch, took about 210 minutes, which is why the release timeout had to go up to 240 minutes (#1343, #1347). Almost all of that was one step,Build CLI. It built the CLI and then ran every test suite one after another, on a 2-vCPU agent.This PR runs the tests in parallel jobs, the same lanes GitHub Actions runs since #1378:
Buildjob no longer runs tests. It builds, packages, signs and publishes exactly as before.Buildstage (.pipelines/templates/test-jobs.yaml): CLI shards 1, 2 and 4, UI Automation, Node/analyzer/Pester, and DevTools native tests. They runbuild-cli.ps1 -OnlyTests -UseExistingArtifacts, the same as GitHub. Every later stage depends on the wholeBuildstage, so a failing test still blocks every release stage.BuildForTestsjob runs the same stable, unstubbed build, unsigned, next toBuild. The tests use its output, so they don't wait for signing. 1ES uploads a job's artifacts only after the whole job ends.Buildjob still runs the find-ui tests (263 tests, about 3 min) on the corpus it refreshes from GitHub and ships. The test jobs build from a fresh checkout, so they only see the committed corpus.Buildjob, which builds what we ship, is still traced. Before, CodeQL traced the test runs too, and timing-sensitive tests failed under it.Build240 → 120 min, releaseBuild240 → 180 min, each test job 90 min.Before / after
Measured on this branch, against
mainruns of the same code (this PR changes only.pipelines/). Nothing was signed or published: on a feature branch, ESRP signing is off and every publishing stage is skipped.mainrun 20261120.6; median 104 over Nov 14–20)Buildjob under CodeQL tracingBuild CLIalone took 149 min, then failed on timing tests (20261117.2)Buildjob took 95 min and passed (20261120.1)Every suite runs, with the same test counts as
mainon the same code:Not tested / risks
rel/v*release. The signing steps and their inputs are unchanged, but the first real release is the first signed run of the new layout.Azure-Pipelines-1ESPT-ExDSharedpool: CI ~107 → ~182, releaseBuildstage ~133 → ~263. Each run uses ~7 agents at once instead of 1. Median wait for an agent was 2 min across 79 runs.DeploymentStateStoreTests.Commit_ConcurrentWritersCannotPublishTheSameRevision, failed in 1 of 7 branch CI runs. That's unchanged frommain.Checklist
scripts\tests\DevToolsEngine.Tests.ps1release-archive gate (readsrelease.yml): 4 passed, 0 failed..pipelines/README.md)Notes for reviewers
Build's signed artifacts. That makes the test path run after signing and SDL, so a release would take ~105 min instead of ~85.BuildForTestsfailed BinSkim BA2007 on the third-partylibHarfBuzzSharp.dll. Its baseline in.config/guardian/.gdnbaselinesmatches files by their path under$(Build.ArtifactStagingDirectory). The job now publishes from there, likebuild.yaml.PackageCommandTests+PerfFixtureTests) is now the slowest lane, at ~28 min. Splitting it further is a follow-up.