[SDK] Ship per-platform native binaries via optional-dependency packages (#512)

.azure-pipelines/1ES.Release.yml

@@ -58,15 +58,43 @@ extends:
           type: releaseJob
           isProduction: true
           inputs:
+          - input: pipelineArtifact
+            pipeline: MXC
+            targetPath: '$(Pipeline.Workspace)/platform-packages'
+            artifactName: mxc-npm-sdk-platform-packages
           - input: pipelineArtifact
             pipeline: MXC
             targetPath: '$(Pipeline.Workspace)/packages'
             artifactName: mxc-npm-sdk-package
 
         displayName: Publish to NPM
         steps:
+        # Publish the per-platform binary packages FIRST. The meta package pins
+        # them as exact-version optionalDependencies, so they must exist on the
+        # registry before the meta package is published.
+        - task: EsrpRelease@10
+          displayName: 'Publish platform packages to NPM'
+          inputs:
+            connectedservicename: ${{ parameters.ESRPInfo.serviceName }}
+            usemanagedidentity: false
+            keyvaultname: ${{ parameters.ESRPInfo.azureKeyVaultName }}
+            authcertname: ${{ parameters.ESRPInfo.authCertName }}
+            signcertname: ${{ parameters.ESRPInfo.signCertName }}
+            clientid: ${{ parameters.ESRPInfo.clientId }}
+            intent: 'PackageDistribution'
+            contenttype: npm
+            contentsource: 'Folder'
+            folderlocation: '$(Pipeline.Workspace)/platform-packages'
+            owners: ${{ parameters.ESRPInfo.OwnersEmail }}
+            approvers: ${{ parameters.ESRPInfo.ApproversEmail }}
+            waitforreleasecompletion: true
+            serviceendpointurl: 'https://api.esrp.microsoft.com'
+            mainpublisher: ESRPRELPACMAN
+            domaintenantid: ${{ parameters.ESRPInfo.tenantId }}
+
+        # Publish the meta package LAST, once the platform packages are live.
         - task: EsrpRelease@10
-          displayName: 'Publish to NPM'
+          displayName: 'Publish meta package to NPM'
           inputs:
             connectedservicename: ${{ parameters.ESRPInfo.serviceName }}
             usemanagedidentity: false

.azure-pipelines/README.md

@@ -22,4 +22,66 @@ from crates.io and npmjs, helping ensure secure and vetted consumption of third
   developer iteration.
 - The ADO pipeline can also be triggered on PRs via `/azp run`
   (see [docs/pull-requests.md](../docs/pull-requests.md)) when reviewers want
-  to run the official build against a change before merge.
+  to run the official build against a change before merge.
+
+## npm SDK packaging: meta + per-platform binary packages
+
+`@microsoft/mxc-sdk` is a **meta package that ships no native binaries**. Each
+host's executor binaries are delivered through one of five per-platform
+packages, which the meta package lists as exact-pinned `optionalDependencies`:
+
+```
+@microsoft/mxc-sdk-win32-x64     @microsoft/mxc-sdk-win32-arm64
+@microsoft/mxc-sdk-linux-x64     @microsoft/mxc-sdk-linux-arm64
+                                 @microsoft/mxc-sdk-darwin-arm64
+```
+
+(Intel macOS / `darwin-x64` is intentionally not shipped — it has no build
+target and no executor has ever shipped for it.)
+
+npm's `os`/`cpu` filtering installs only the package matching the consuming
+host, so an install downloads just that host's payload.
+
+### Pipeline flow
+
+- **Build** (`Build_Binaries`) produces the `wxc-binaries-*` / `lxc-binaries-*` /
+  `mxc-binaries-*` artifacts.
+- **Package** (`Package.NpmSdk.Job.yml`) stages each artifact into
+  `sdk/platform-packages/<os>-<arch>/`, verifies versions/pins are in sync
+  (`scripts/sync-platform-package-versions.js --check`), packs the meta package
+  (artifact `mxc-npm-sdk-package`) and each per-platform package whose primary
+  binary is present (artifact `mxc-npm-sdk-platform-packages`), then runs a
+  **release-completeness gate** that fails the build unless every
+  `@microsoft/mxc-sdk-*` optional dependency the meta package pins has a matching
+  packed tarball.
+- **Release** (`1ES.Release.yml`) publishes the **platform packages first**, then
+  the **meta package last** — the meta's exact-pinned optional deps must already
+  exist on the registry when it is published.
+
+### One-time / ops setup (manual — not automated here)
+
+- **Register the five package names** under the `@microsoft` npm org with the same
+  access, provenance, and signing settings as `@microsoft/mxc-sdk`
+  (`@microsoft/mxc-sdk-{win32,linux,darwin}-{x64,arm64}` minus `darwin-x64`,
+  which is not shipped). The release step cannot create org-scoped names that do
+  not yet exist with the right permissions. **Recommended preflight:** before the
+  meta publish, verify each name exists under the expected org/provenance and that
+  the exact version is either being published this run or already present.
+- **Lockfile integrity is bootstrapped at first publish.** Until the platform
+  packages exist on the registry, `sdk/package-lock.json` records them as stubs
+  with no `resolved`/`integrity` (you cannot hash an unpublished tarball). After
+  the **first** successful publish, run `npm install --package-lock-only` in
+  `sdk/` and commit the lockfile so the optional-dep entries gain
+  `resolved`/`integrity`; subsequent `npm ci` then verifies the native-binary
+  tarball hashes.
+- **Partial-publish recovery.** Platform-first / meta-last is two ESRP steps and
+  is not atomic: if it fails after publishing some platform packages, those
+  versions are immutable (re-publishing the same version returns `403`). Recover
+  by bumping the version (`src/Cargo.toml` + `sdk/package.json`, then
+  `node scripts/sync-platform-package-versions.js`) and re-running the release.
+- **Offline / air-gapped mirrors** must carry all five platform packages, not just
+  the meta package.
+- **Cross-compiled consumers** (e.g. building a Windows VS Code bundle on a Linux
+  agent) get the *agent's* platform package from a plain `npm install`. Such
+  consumers must use npm's `--os`/`--cpu` install overrides (npm 10+) or a
+  packaging step that force-installs the target platform package.

.azure-pipelines/templates/1ES.Build.Stages.yml

@@ -68,14 +68,19 @@ stages:
     parameters:
       targets:
         - artifact: wxc-binaries-x86_64-pc-windows-msvc
+          os: win32
           sdkArch: x64
         - artifact: wxc-binaries-aarch64-pc-windows-msvc
+          os: win32
           sdkArch: arm64
         - artifact: lxc-binaries-x86_64-unknown-linux-gnu
+          os: linux
           sdkArch: x64
         - artifact: lxc-binaries-aarch64-unknown-linux-gnu
+          os: linux
           sdkArch: arm64
         - artifact: mxc-binaries-aarch64-apple-darwin
+          os: darwin
           sdkArch: arm64
 
       ESRPInfo: ${{ parameters.ESRPInfo }}

.azure-pipelines/templates/Package.NpmSdk.Job.yml

@@ -6,12 +6,16 @@ parameters:
   type: object
   default:
     - artifact: wxc-binaries-x86_64-pc-windows-msvc
+      os: win32
       sdkArch: x64
     - artifact: wxc-binaries-aarch64-pc-windows-msvc
+      os: win32
       sdkArch: arm64
     - artifact: lxc-binaries-x86_64-unknown-linux-gnu
+      os: linux
       sdkArch: x64
     - artifact: lxc-binaries-aarch64-unknown-linux-gnu
+      os: linux
       sdkArch: arm64
 - name: ESRPInfo
   type: object
@@ -27,7 +31,11 @@ jobs:
   variables:
     outputDirectory: $(Build.SourcesDirectory)/out
     sdkDirectory: $(Build.SourcesDirectory)/sdk
+    platformPackagesDir: $(Build.SourcesDirectory)/sdk/platform-packages
+    # Artifact carrying the meta @microsoft/mxc-sdk tarball (no binaries).
     artifactBaseName: mxc-npm-sdk-package
+    # Artifact carrying the five per-platform binary package tarballs.
+    platformArtifactName: mxc-npm-sdk-platform-packages
     # Cache npm package downloads across runs to avoid re-fetching every dependency
     # tarball from the Azure Artifacts feed on each pipeline invocation.
     npm_config_cache: $(Pipeline.Workspace)/.npm
@@ -48,32 +56,31 @@ jobs:
           npm | "$(Agent.OS)"
         path: $(npm_config_cache)
 
-    # Download all artifacts dynamically
+    # Download each platform's binaries straight into its platform package
+    # directory: sdk/platform-packages/<os>-<arch>/.
     - ${{ each target in parameters.targets }}:
       - task: DownloadPipelineArtifact@2
         displayName: Download ${{ target.artifact }}
         inputs:
           artifact: ${{ target.artifact }}
-          path: $(sdkDirectory)/bin/${{ target.sdkArch }}
+          path: $(platformPackagesDir)/${{ target.os }}-${{ target.sdkArch }}
 
     # Pipeline artifact downloads strip Unix execute permissions.
-    # Restore +x on Linux and macOS binaries so the tarball preserves them.
-    - script: chmod +x $(sdkDirectory)/bin/*/lxc-exec
-      displayName: Restore execute permission on lxc-exec
-
+    # Restore +x on Linux and macOS binaries so the tarballs preserve them.
     - script: |
         set -euo pipefail
-        find $(sdkDirectory)/bin -name linux-test-proxy -exec chmod +x {} +
-      displayName: Restore execute permission on linux-test-proxy
-
-    # Drop symbol files from the npm package — symbols ship separately.
-    - script: rm -rf $(sdkDirectory)/bin/*/symbols
-      displayName: Strip symbols/ from npm package
+        find $(platformPackagesDir) -name lxc-exec -exec chmod +x {} +
+        find $(platformPackagesDir) -name mxc-exec-mac -exec chmod +x {} +
+      displayName: Restore execute permission on native binaries
 
+    # Symbols ship separately, and dev/test-only proxies are not shipped in the
+    # per-platform packages.
     - script: |
         set -euo pipefail
-        find $(sdkDirectory)/bin -name mxc-exec-mac -exec chmod +x {} +
-      displayName: Restore execute permission on mxc-exec-mac
+        rm -rf $(platformPackagesDir)/*/symbols
+        rm -f $(platformPackagesDir)/*/wxc-test-proxy.exe
+        rm -f $(platformPackagesDir)/*/linux-test-proxy
+      displayName: Strip symbols and dev/test-only binaries
 
     # Copy .npmrc to the SDK directory so Azure artifacts feed is used for npm install.
     - task: CopyFiles@2
@@ -96,20 +103,64 @@ jobs:
       workingDirectory: $(sdkDirectory)
       displayName: npm run build
 
-    - script: npm pack
+    # Verify platform package versions and pins are in sync before packing
+    # (CI must not silently mutate tracked files — fail on drift instead).
+    - script: node scripts/sync-platform-package-versions.js --check
+      workingDirectory: $(Build.SourcesDirectory)
+      displayName: Check platform package versions
+
+    # Pack the meta package (no native binaries) into out/packages.
+    - script: |
+        set -euo pipefail
+        mkdir -p $(outputDirectory)/packages
+        npm pack --pack-destination $(outputDirectory)/packages
       workingDirectory: $(sdkDirectory)
-      displayName: npm pack
+      displayName: npm pack (meta package)
 
-    - task: CopyFiles@2
-      displayName: Copy npm package
-      inputs:
-        sourceFolder: $(sdkDirectory)
-        contents: |
-          *.tgz
-        targetFolder: '$(outputDirectory)/packages'
+    # Pack each per-platform package, gated on the presence of its expected
+    # primary executor (an allowlist, not an "any extra file" denylist), so a
+    # stray file can never cause a broken/partial package to be packed.
+    - script: |
+        set -euo pipefail
+        mkdir -p $(outputDirectory)/platform-packages
+        for d in $(platformPackagesDir)/*/; do
+          name=$(basename "$d")
+          case "$name" in
+            win32-*)  primary="wxc-exec.exe" ;;
+            linux-*)  primary="lxc-exec" ;;
+            darwin-*) primary="mxc-exec-mac" ;;
+            *) echo "Unknown platform package dir: $name" >&2; exit 1 ;;
+          esac
+          if [ -f "$d/$primary" ]; then
+            (cd "$d" && npm pack --pack-destination $(outputDirectory)/platform-packages)
+            echo "Packed platform package: $name"
+          else
+            echo "Skipping $name: primary binary $primary not staged (not built this run)"
+          fi
+        done
+        ls -la $(outputDirectory)/platform-packages
+      displayName: npm pack (platform packages)
+
+    # Release-completeness gate. The on-disk platform-packages dir is the
+    # canonical source: the set must be non-empty, the meta package must pin
+    # exactly those packages at their versions, every expected platform tarball
+    # must be packed, and NO stray/extra .tgz (mxc or otherwise) may be present.
+    - script: |
+        set -euo pipefail
+        META_PKG="$(sdkDirectory)/package.json" \
+        PLATFORM_PACKAGES_DIR="$(sdkDirectory)/platform-packages" \
+        TARBALL_DIR="$(outputDirectory)/platform-packages" \
+        node "$(Build.SourcesDirectory)/scripts/check-release-completeness.js"
+      displayName: Verify release completeness (canonical platform-packages set)
 
     - task: 1ES.PublishPipelineArtifact@1
-      displayName: Upload SDK as pipeline artifact
+      displayName: Upload meta SDK package
       inputs:
         path: '$(outputDirectory)/packages'
-        artifactName: $(artifactBaseName)
+        artifactName: $(artifactBaseName)
+
+    - task: 1ES.PublishPipelineArtifact@1
+      displayName: Upload platform binary packages
+      inputs:
+        path: '$(outputDirectory)/platform-packages'
+        artifactName: $(platformArtifactName)

.azure-pipelines/templates/Rust.Build.Job.yml

@@ -188,6 +188,41 @@ jobs:
           contents: wslcsdk.dll
           targetFolder: $(outputDirectory)/$(targetTriple)
 
+      # Stage the FULL shipped payload from the platform package's `files`
+      # allowlist (the single source of truth) — most importantly the win32-x64
+      # micro-VM payload (nanvixd.exe, nanvix_rootfs.img, python3.initrd,
+      # bin/kernel.elf, snapshots/*) that `signPattern` does NOT cover —
+      # preserving the bin/ + snapshots/ substructure, and HARD-FAIL if any
+      # allowlisted artifact is missing. `npm pack` silently omits absent
+      # `files[]` entries, so without this the official build would publish a
+      # binary-deficient package that the name-only checks never catch. Reads the
+      # manifest with ConvertFrom-Json (no Node needed on the Rust build agent),
+      # mirroring the manifest-driven staging in Build.Windows.Job.yml.
+      - task: PowerShell@2
+        displayName: Stage shipped payload (manifest-driven)
+        condition: and(succeeded(), eq('${{ item.os }}', 'windows'))
+        inputs:
+          targetType: inline
+          script: |
+            $tuple = if ('$(targetTriple)' -eq 'x86_64-pc-windows-msvc') { 'win32-x64' } else { 'win32-arm64' }
+            $manifest = "$(Build.SourcesDirectory)/sdk/platform-packages/$tuple/package.json"
+            $src = "$(targetTripleDir)"
+            $dest = "$(outputDirectory)/$(targetTriple)"
+            $files = (Get-Content -Raw $manifest | ConvertFrom-Json).files | Where-Object { $_ -ne 'README.md' }
+            $missing = @()
+            foreach ($f in $files) {
+              $from = Join-Path $src $f
+              if (-not (Test-Path $from)) { $missing += $f; continue }
+              $to = Join-Path $dest $f
+              New-Item -ItemType Directory -Force -Path (Split-Path $to) | Out-Null
+              Copy-Item $from $to -Force
+            }
+            if ($missing.Count -gt 0) {
+              Write-Error "Missing shipped payload file(s) for ${tuple}: $($missing -join ', ')"
+              exit 1
+            }
+            Write-Host "Staged $($files.Count) payload file(s) for $tuple."
+
       # Copy symbol files into a sibling `symbols/` subdir, NOT alongside the
       # binaries. Workaround for BinSkim BA2007: when the pdb is co-located,
       # BinSkim reads its S_COMPILE3 records and flags `-wd4146` in the