Skip to content

Cut ADO CI from about 105 to 68 minutes and releases from 141 to about 90 - #1384

Open
Nikola Metulev (nmetulev) wants to merge 6 commits into
mainfrom
nmetulev-speed-up-ado-pipelines
Open

Nikola Metulev (nmetulev) wants to merge 6 commits into
mainfrom
nmetulev-speed-up-ado-pipelines

Conversation

@nmetulev

@nmetulev Nikola Metulev (nmetulev) commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

What and why

The internal Azure DevOps pipelines took about 105 minutes for CI and 141 minutes for a release. A Monday release rehearsal, or the first run of a rel/v* branch, took about 210 minutes, which is why the release timeout had to go up to 240 minutes (#1343, #1347). Almost all of that was one step, Build CLI. It built the CLI and then ran every test suite one after another, on a 2-vCPU agent.

This PR runs the tests in parallel jobs, the same lanes GitHub Actions runs since #1378:

  • The Build job no longer runs tests. It builds, packages, signs and publishes exactly as before.
  • New test jobs in the Build stage (.pipelines/templates/test-jobs.yaml): CLI shards 1, 2 and 4, UI Automation, Node/analyzer/Pester, and DevTools native tests. They run build-cli.ps1 -OnlyTests -UseExistingArtifacts, the same as GitHub. Every later stage depends on the whole Build stage, so a failing test still blocks every release stage.
  • Release only: a new BuildForTests job runs the same stable, unstubbed build, unsigned, next to Build. The tests use its output, so they don't wait for signing. 1ES uploads a job's artifacts only after the whole job ends.
  • Release only: the Build job still runs the find-ui tests (263 tests, about 3 min) on the corpus it refreshes from GitHub and ships. The test jobs build from a fresh checkout, so they only see the committed corpus.
  • The test jobs opt out of CodeQL, with a justification. The Build job, which builds what we ship, is still traced. Before, CodeQL traced the test runs too, and timing-sensitive tests failed under it.
  • Timeouts: CI Build 240 → 120 min, release Build 240 → 180 min, each test job 90 min.

Before / after

Measured on this branch, against main runs of the same code (this PR changes only .pipelines/). Nothing was signed or published: on a feature branch, ESRP signing is off and every publishing stage is skipped.

Before After Saving
ADO CI, queue → finish 102.6 min (main run 20261120.6; median 104 over Nov 14–20) 65.6 / 67.7 / 67.9 / 68.7 min (4 of 4 runs green) −35 min
Release, queue → finish 141.2 min (rel/v0.8.0, 20261106.5) 82.3 / 84.9 min (20261120.5, .3; publishing skipped, a real release adds ~6) −56 min
Release Build job under CodeQL tracing Build CLI alone took 149 min, then failed on timing tests (20261117.2) whole Build job took 95 min and passed (20261120.1) ≥ −55 min

Jobs on the timeline before and after, for CI and a release

Every suite runs, with the same test counts as main on the same code:

Suite Before (one job) After (per job)
WinApp.Cli.Tests 10,384 115 + 9,144 + 50 + 1,075 = 10,384
UI Automation / analyzer / npm 1,085 / 92 / 350 1,085 / 92 / 350
NuGet / scripts Pester 140 / 365 140 / 365
DevTools native (window checks) 14,360 14,360

Not tested / risks

  • Signing is not exercised. It only runs on a real rel/v* release. The signing steps and their inputs are unchanged, but the first real release is the first signed run of the new layout.
  • The test jobs opt out of CodeQL, so test projects are no longer in the CodeQL database. Shipped code is still traced. 1ES documents this opt-out per job, with a justification.
  • About 2× agent-minutes per run on the shared Azure-Pipelines-1ESPT-ExDShared pool: CI ~107 → ~182, release Build stage ~133 → ~263. Each run uses ~7 agents at once instead of 1. Median wait for an agent was 2 min across 79 runs.
  • The known ADO-only flaky test, DeploymentStateStoreTests.Commit_ConcurrentWritersCannotPublishTheSameRevision, failed in 1 of 7 branch CI runs. That's unchanged from main.

Checklist

  • Tested on Azure DevOps: 7 CI runs and 4 release runs of this branch, plus both pipelines compiled with the ADO preview API.
  • scripts\tests\DevToolsEngine.Tests.ps1 release-archive gate (reads release.yml): 4 passed, 0 failed.
  • Docs updated (.pipelines/README.md)
Notes for reviewers
  • Why a second, unsigned build in the release: the test jobs could instead wait for Build's signed artifacts. That makes the test path run after signing and SDL, so a release would take ~105 min instead of ~85.
  • Run 20261120.1's BuildForTests failed BinSkim BA2007 on the third-party libHarfBuzzSharp.dll. Its baseline in .config/guardian/.gdnbaselines matches files by their path under $(Build.ArtifactStagingDirectory). The job now publishes from there, like build.yaml.
  • CLI shard 1 (PackageCommandTests + PerfFixtureTests) is now the slowest lane, at ~28 min. Splitting it further is a follow-up.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… baselines match

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 23:20
@nmetulev Nikola Metulev (nmetulev) added the agent-preparing Agent is addressing feedback or completing required validation and CI label Nov 20, 2026
…t results re-runnable

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The shipping build must test the baked snapshot, since the current test gate checks the committed snapshot rather than the data included in the CLI.

1 open issue
What changed in this PR

Shortens Azure DevOps CI and release waits by moving tests out of the shipping build and into parallel jobs, following the repository’s GitHub Actions test layout.

Changes:

  • Add parallel CLI shards, UI Automation, Auxiliary, and DevTools native test jobs with separate result artifacts.
  • Add an unsigned release build for testing and disable CodeQL tracing in release test-only jobs.
  • Skip test suites in the shipping build, reduce job timeouts, and document the new layout.
File Description
.pipelines/​templates/​test-jobs.yaml Define shared test lanes, artifact downloads, and result uploads.
.pipelines/​templates/​build.yaml Skip in-job tests and remove centralized test-result publishing.
.pipelines/​release.yml Add an unsigned test build and parallel lanes; reduce the shipping build timeout.
.pipelines/​README.md Document test jobs, artifact sources, and troubleshooting.
.pipelines/​ci.yml Connect parallel test jobs to build artifacts and lower the build timeout.

Review effort: Balanced

Comment thread .pipelines/templates/build.yaml Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The failure-triage instructions incorrectly direct all failed lanes to .trx files, although Pester results use XML and npm and native-unit failures appear in job logs.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Low severity Failed lanes may report results outside .trx files

.pipelines/​README.md:230

Optional — This directs every failed lane to a .trx file, but the Auxiliary lane writes Pester results as nuget-pester.xml and scripts-pester.xml; npm failures and native-unit failures are reported in the job log instead. Please point readers to the logs and the available report formats.

1 resolved since last review

Review effort: Balanced

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Build Metrics Report

Validation passed. All required build and validation jobs succeeded.

Binary Sizes

Artifact Baseline Current Delta
CLI (ARM64) 67.56 MB 67.56 MB ✅ 0.0 KB (0.00%)
CLI (x64) 67.41 MB 67.41 MB ✅ 0.0 KB (0.00%)
MSIX (ARM64) 27.88 MB 27.88 MB 📉 -0.2 KB (-0.00%)
MSIX (x64) 29.56 MB 29.56 MB 📉 -0.2 KB (-0.00%)
NPM Package 58.06 MB 58.06 MB 📉 -0.2 KB (-0.00%)
NuGet Package 58.23 MB 58.23 MB 📉 -0.5 KB (-0.00%)

.NET Test Results (TRX reports)

Other suites are reflected in the overall validation status above.

✅ 11425 passed, 44 skipped out of 11469 tests in 1612.2s (-94.9s vs. baseline)

Test Coverage

✅ 87.4% line coverage, 81.5% branch coverage · ✅ no change vs. baseline

CLI Startup Time

72ms median (x64, winapp --version) · ⚠️ +18ms vs. baseline


Updated 2026-11-21 01:24:15 UTC · commit 935a970 · workflow run

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nmetulev Nikola Metulev (nmetulev) added ready-for-review Agent work and technical checks complete; awaiting review or re-review, not approval or merge and removed agent-preparing Agent is addressing feedback or completing required validation and CI labels Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-review Agent work and technical checks complete; awaiting review or re-review, not approval or merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants