Skip to content

Add sparse VHD support to WSLC - #41742

Draft
Ben Hillis (benhillis) wants to merge 19 commits into
masterfrom
sparse-vhd-wslc
Draft

Ben Hillis (benhillis) wants to merge 19 commits into
masterfrom
sparse-vhd-wslc

Conversation

@benhillis

@benhillis Ben Hillis (benhillis) commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Summary of the Pull Request

  • Adds Sparse as an experimental, mutually exclusive WSLC VHD type in the C and WinRT SDKs.
  • Supports sparse VHDs for session storage and named volumes, including session.storageType: sparse (default and dynamic are also accepted).
  • Documents the API contract while keeping usage examples on dynamic VHDs.
  • Adds C SDK, WinRT, service validation, settings, and runtime-version coverage.

This follows #41726, which added the underlying sparse VHD creation support.

PR Checklist

  • Closes: No linked issue.
  • Communication: I've discussed this with core contributors already. If work hasn't been agreed, this work might be rejected
  • Tests: Added/updated if needed and all pass
  • Localization: All end user facing strings can be localized
  • Dev docs: Added/updated if needed
  • Documentation updated: If checked, please file a pull request on our docs repo and link it here: Not submitted.

Detailed Description of the Pull Request / Additional comments

Sparse session storage and named volumes require WSL 3.1.0, using the same checker and minimum version as nested virtualization (#41257) and large host backing pages (#42110). Older releases return the existing WSLC_E_RUNTIME_UPDATE_NEEDED / WinRT RuntimeUpdateNeeded with a localized installed/required-version message. Matching SDK/runtime development builds remain supported. Session handles retain their runtime version for later named-volume operations.

Rebased onto current master, preserving host-loopback support and the newer compatibility interfaces. Sparse remains public VHD type 2 and internal storage flag 0x04, the highest allocated storage bit. It does not consume a FeatureFlags bit: nested virtualization uses 0x40 and large pages use 0x80.

Fixed and sparse named-volume options are mutually exclusive. The SDK preserves client-side error details, and session handles must be constructed with a known runtime version.

The default-session YAML setting is session.storageType, with accepted values default, dynamic, and sparse. default and dynamic currently use standard dynamically expanding storage, while sparse opts newly created session storage into sparse VHD creation. Existing storage is unchanged.

Validation Steps Performed

  • cmake --build . -- -m - passed: full Windows x64 Debug build following a clean, including integration-test sources.
  • .\FormatSource.ps1 -Branch -BaseBranch origin/master -Verify $true - passed.
  • python -m mkdocs build -f doc\mkdocs.yml - passed with existing navigation notices.
  • python tools\devops\validate-localization.py - passed with expected downstream-translation warnings.
  • git -c core.whitespace=cr-at-eol diff --check - passed.
  • & 'Q:\src\WSL\packages\Microsoft.Taef.10.100.251104001\build\Binaries\x64\te.exe' ($out + '.dll') - passed: 41/41 isolated tests (2 runtime-version tests and 39 settings tests). The DLL contains WslcRuntimeVersionTests.obj and WSLCSettingsUnitTests.obj, linked with the existing test libraries and without host-changing module setup. Covers version boundaries, localized diagnostics, development-build matching, sparse setting parsing, invalid values, and defaults.
  • python -X utf8 tools\devops\validate-copyright-headers.py - reports missing headers in generated/dependency files; the repository validator's separate changed-source check passes.

Earlier validation, before this rebase:

  • cmake --build . -- -m - passed: full x64 Debug build.
  • .\FormatSource.ps1 - passed.
  • bin\x64\Debug\test.bat /name:*WSLCCLISettingsUnitTests* - passed: 39/39.
  • bin\x64\Debug\test.bat /name:*WslcSdkTests::SparseVhdRequiresWslUpdate* - passed: 1/1.
  • bin\x64\Debug\test.bat /name:*WslcSdkTests::SessionCreateVhd* - passed: 1/1.
  • bin\x64\Debug\test.bat /name:*WslcSdkWinRtTests::SessionCreateVhd* - passed: 1/1.
  • python -m mkdocs build -f doc\mkdocs.yml - passed with existing navigation notices.
  • python tools\devops\validate-localization.py - passed with expected downstream-translation warnings.

Live older-runtime rejection and sparse-VHD creation/attachment integration tests have now passed after this rebase, as recorded below.

Cross-PR validation: Combined all three rebased features locally, resolving overlapping declarations and session gates. cmake . and cmake --build . --clean-first -- -m passed. The same TAEF command above passed 44/44 tests with the runtime, memory-policy, and settings objects linked together. A manual C SDK settings probe preserved nested (0x40), large-page (0x80), and combined flags, with and without GPU, alongside sparse storage settings without clobbering either field. No session or VM was created by the probe.

The final sparse older-runtime regression test covers 3.0.2 and 3.0.5, excluding an exact match to the SDK version because matching development builds intentionally bypass the release minimum. Its final form was compiled in the combined full-project build (cmake --build . -- -m - passed) and passed in the live integration run below.

Live cross-PR validation: Deployed a clean full build containing all three features and the nested module-loading startup fix. SDK/runtime development version 3.0.5.6 matched, exercising the intentional development-build exception. Ran this exact TAEF command line from an isolated working directory, without a setup script or distribution reimport:

Q:\src\WSL\packages\Microsoft.Taef.10.100.251104001\build\Binaries\x64\te.exe Q:\src\WSL\bin\x64\Debug\wsltests.dll /p:Version=2 /p:DistroPath=Q:\src\WSL\packages\Microsoft.WSL.TestDistro.2.7.1-1\x64\test_distro.tar.xz /p:TestDataPath=Q:\src\WSL\packages\Microsoft.WSL.TestData.0.6.0\x64 /p:PullRequest=False /p:AllowUnsigned=1 /select:"@Name='*LargeMemoryBackingPages*' or @Name='*ContainerNestedVirtualization' or @Name='*SessionSettingsNestedVirtualization' or @Name='*WSLContainerNestedVirtualizationDisabled' or @Name='*SessionCreateVhd' or @Name='*SparseVhdRequiresWslUpdate' or @Name='*WSLCTests::Modules' or @Name='*RootNamespaceProcessReleaseAndExitStress' or @Name='*WslcRuntimeVersionTests*' or @Name='*MemoryBackingPagesTests*' or @Name='*WSLCCLISettingsUnitTests*'"

Passed: 55/55, no skips. Includes C/WinRT sparse session storage and named-volume creation/attachment with sparse-file attribute assertions, fixed/dynamic compatibility, cached older-runtime rejection, settings/version tests, nested KVM/policy/process-reaping tests, and WSL/C SDK/COM large-page reporting.

A separate manual C SDK probe passed five create/load-image/terminate cycles with nested (0x40) and large-page (0x80) flags combined with sparse session storage; each initialized storage VHD had FILE_ATTRIBUTE_SPARSE_FILE. Before upgrading, a live probe against installed WSL 3.0.4 rejected nested, large, sparse, and combined requests with WSLC_E_RUNTIME_UPDATE_NEEDED, installed/required version diagnostics, no session handle, and no storage creation.

Original default distribution and .wslconfig were restored. Historical Windows hosts and physical host backing-page allocation/performance were not tested.

Copilot AI balanced review requested due to automatic review settings September 30, 2026 17:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

It changes public SDK surfaces and storage-creation behavior across multiple process boundaries, warranting final human review.

Review effort: Balanced
Findings: None

What changed in this PR

Adds experimental sparse VHD support across WSLC’s SDK, service, settings, documentation, and tests.

Changes:

  • Adds sparse VHD types and propagates them through session and volume creation.
  • Adds the session.sparseVhd default-session setting.
  • Documents and tests sparse VHD behavior across C and WinRT APIs.
File Description
test/​windows/​WSLCTests.cpp Updates invalid storage-flag validation.
test/​windows/​WslcSdkWinRTTests.cpp Tests sparse session and volume VHDs via WinRT.
test/​windows/​WslcSdkTests.cpp Tests sparse VHDs through the C SDK.
test/​windows/​wslc/​WSLCCLISettingsUnitTests.cpp Covers sparse setting parsing.
src/​windows/​wslcsession/​WSLCVhdVolume.cpp Parses and applies sparse volume options.
src/​windows/​wslcsession/​WSLCSession.cpp Creates sparse session storage when requested.
src/​windows/​WslcSDK/​wslcsdk.h Exposes the C sparse VHD type.
src/​windows/​WslcSDK/​wslcsdk.cpp Maps sparse SDK options into service settings.
src/​windows/​WslcSDK/​winrt/​wslcsdk.idl Exposes the WinRT sparse VHD type.
src/​windows/​service/​inc/​WSLCShared.idl Adds the sparse session-storage flag.
src/​windows/​service/​exe/​WSLCSessionManager.cpp Applies the default-session sparse setting.
src/​windows/​common/​wslc/​settings/​WSLCUserSettings.h Defines the new setting.
src/​windows/​common/​wslc/​settings/​WSLCUserSettings.cpp Documents and validates the setting.
doc/​docs/​api-reference/​csharp/​enumerations/​vhdtype.md Documents C# sparse VHD semantics.
doc/​docs/​api-reference/​cpp/​enumerations/​vhdtype.md Documents C++ sparse VHD semantics.
doc/​docs/​api-reference/​c/​structures/​wslcvhdrequirements.md Documents sparse requirements support.
doc/​docs/​api-reference/​c/​storage-apis/​wslccreatesessionvhdvolume.md Documents sparse named volumes.
doc/​docs/​api-reference/​c/​session-apis/​wslcsetsessionsettingsvhd.md Documents sparse session storage.
doc/​docs/​api-reference/​c/​enumerations/​wslcvhdtype.md Documents the C enum value.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@benhillis
Ben Hillis (benhillis) marked this pull request as ready for review September 30, 2026 19:18
Comment thread src/windows/WslcSDK/wslcsdk.cpp Resolved
Copilot AI balanced review requested due to automatic review settings October 1, 2026 20:25
Comment thread src/windows/WslcSDK/wslcsdk.cpp Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The WinRT error surface omits the new runtime-update error, and key parser and settings integration paths lack coverage.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Previously missed (2)

In code that hasn't changed since last review

Low severity Add end-to-end test for session sparse VHD creation

src/​windows/​service/​exe/​WSLCSessionManager.cpp:102

The added settings test only proves that YAML parses to SessionSparseVhd; it does not exercise this service wiring. Add an end-to-end settings test that enables session.sparseVhd, creates a fresh default-session storage VHD, and verifies FILE_ATTRIBUTE_SPARSE_FILE, so regressions between parsing, flag propagation, and VHD creation are covered.

Low severity Test malformed Sparse values and Fixed/Sparse conflict

src/​windows/​wslcsession/​WSLCVhdVolume.cpp:56

The existing NamedVolumeVhdOptionsParseTest exhaustively covers malformed Fixed values and option combinations, but no test covers malformed Sparse values or the new Fixed=true/Sparse=true rejection. Add both cases to lock down this new parser contract and mutual-exclusion check.

Comment thread src/windows/WslcSDK/winrt/wslcsdk.idl Resolved
Copilot AI balanced review requested due to automatic review settings October 1, 2026 20:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Compatibility gating, parser rejection, and settings propagation need direct automated coverage.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Previously missed (3)

In code that hasn't changed since last review

Low severity Add runtime-version gate coverage

src/​windows/​WslcSDK/​wslcsdk.cpp:315

The new runtime-version gate is not exercised by the added SDK tests: those tests run client and service from the same build, so currentVersion == clientVersion always bypasses the minimum-version rejection. Add focused coverage for versions below, equal to, and above 3.0.2, plus the lockstep-development exception, so the documented WSLC_E_WSL_UPDATE_NEEDED contract cannot regress unnoticed.

Low severity Test sparse VHD setting propagation and file attribute

src/​windows/​service/​exe/​WSLCSessionManager.cpp:102

The settings test only verifies that session.sparseVhd parses as true; it does not exercise this propagation into default-session storage flags or verify that the resulting storage.vhdx is sparse. Add a fresh-default-session integration test using this setting and assert FILE_ATTRIBUTE_SPARSE_FILE, since otherwise the setting can be silently disconnected while all current tests still pass.

Low severity Add malformed Sparse and conflicting VHD option tests

src/​windows/​wslcsession/​WSLCVhdVolume.cpp:56

The existing exhaustive NamedVolumeVhdOptionsParseTest covers malformed Fixed, but no cases were added for malformed Sparse values or for the new Fixed=true plus Sparse=true rejection. Add both cases so the parser and the promised mutual-exclusion rule are directly covered rather than only testing the valid SDK path.

Copilot AI balanced review requested due to automatic review settings October 1, 2026 20:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The new parser validation and older-runtime compatibility paths lack targeted regression coverage.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Previously missed (2)

In code that hasn't changed since last review

Medium severity Add version predicate tests including lockstep development exception

src/​windows/​WslcSDK/​wslcsdk.cpp:314

The added sparse tests run the SDK and service at the same package version, so the currentVersion != clientVersion exemption prevents this new failure path from ever being exercised. Because the public docs promise WSLC_E_WSL_UPDATE_NEEDED for older runtimes, factor the version predicate into testable logic and cover versions below, equal to, and above 3.0.2 plus the lockstep-development exception.

Medium severity Cover Sparse parsing and Fixed plus Sparse validation

src/​windows/​wslcsession/​WSLCVhdVolume.cpp:56

The new Sparse option parsing and the Fixed && Sparse rejection are not covered by NamedVolumeVhdOptionsParseTest, even though that test already exercises malformed Fixed values (test/windows/WSLCTests.cpp:5195-5201). Add cases for malformed/empty Sparse values and for both options being true so direct service callers' validation contract cannot regress.

OneBlue
Blue (OneBlue) previously approved these changes Oct 1, 2026

@OneBlue Blue (OneBlue) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, minor comment

Comment thread src/windows/WslcSDK/wslcsdk.cpp Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

Sparse behavior is consistently implemented, compatibility-gated, validated, tested, and documented across the affected surfaces.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Copilot AI balanced review requested due to automatic review settings October 2, 2026 13:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 2 Medium severity · 1 Low severity

Open (3)

Comment thread src/windows/WslcSDK/wslcsdk.cpp Resolved
Comment thread src/windows/WslcSDK/wslcsdk.cpp Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The runtime-version failure discards its localized error message before it reaches C and WinRT callers.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Resolved since last review (3)

Comment thread src/windows/WslcSDK/wslcsdk.cpp Resolved
Copilot AI balanced review requested due to automatic review settings October 2, 2026 16:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 2 Medium severity · 2 Low severity

Open (4)

Comment thread src/windows/WslcSDK/wslcsdk.cpp Resolved
Comment thread test/windows/WslcSdkWinRTTests.cpp Resolved
Comment thread test/windows/WslcSdkWinRTTests.cpp Resolved
@Aut0matas

Copy link
Copy Markdown

Not a review — just field data that may be useful here.

On WSL 3.0.2.0 (the 3.0.2 pre-release: has #41726, not this PR), a default wslc session with zero images, zero containers, zero volumes — inside it, /dev/sdc (1 TB ext4 at /var/lib/docker) reports 2.6 MiB used — still has a storage.vhdx of 34.78 GiB, and per fsutil sparse queryflag it is not sparse, so all of that is committed on the system drive. Reading the BAT directly:

BAT state blocks size
FullyPresent 979 0.96 GiB
Unmapped 34,904 34.09 GiB
Zero 16,989 16.59 GiB
NotPresent 995,705 972.37 GiB

34,499 of the Unmapped blocks still carry a file offset: the guest TRIM reached the BAT but the host file space was never returned. fstrim -v reports 0 B trimmed (the mount already carries discard), so there is nothing left to poke from inside the session.

Related, and not documented anywhere I could find: the "Enable sparse VHD by default" toggle in WSL Settings silently does not apply to wslc. It writes [experimental] sparseVhd into .wslconfig, which only affects WSL2 distros. Same machine, same .wslconfig (sparseVhd=true, set weeks before the session in question):

  • %LOCALAPPDATA%\wsl\{guid}\ext4.vhdx → this file is set as sparse
  • a wslc session storage.vhdx created the same day → this file is NOT set as sparse

Sessions aren't registered as distros either, so wsl --manage <name> --set-sparse true returns WSL_E_DISTRO_NOT_FOUND. With that toggle reading On, it is easy to assume the growth is already handled — a pointer to session.storageType: sparse in the settings UI or docs once this lands would save users the digging.

Copilot AI balanced review requested due to automatic review settings October 12, 2026 15:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation consistently covers SDK contracts, compatibility gating, service behavior, configuration, localization, tests, and documentation.

0 open findings

🧠 Review effort: Balanced

Ben Hillis and others added 19 commits October 30, 2026 11:20
Add one minimum-version constant, localized runtime-update diagnostic, and isolated boundary tests for newly introduced SDK features. Preserve matching development-runtime support without changing existing feature compatibility.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c41a7ed8-95e1-43b6-adb5-b4e50c9dbff8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The wrapper is not reused after a failed capture in current SDK paths; keeping the first error could mismatch a later HRESULT.

This reverts commit 1966502.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e2f5e2f3-d12c-4bd3-aa3d-bfc156a97ae2
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e2f5e2f3-d12c-4bd3-aa3d-bfc156a97ae2
Reuse the shared runtime version checker and existing runtime-update error. Keep sparse in StorageFlags bit 0x04, independent of session feature bits, and retain current compatibility interfaces and host loopback support.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c41a7ed8-95e1-43b6-adb5-b4e50c9dbff8
Exercise released pre-3.1 versions without expecting the deliberate matching SDK/runtime development exception to fail.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c41a7ed8-95e1-43b6-adb5-b4e50c9dbff8

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Post-rebase live Windows integration validation remains pending for sparse session and named-volume creation, VHD attachment, and older-runtime rejection.

No open issues 🎉

Review effort: Balanced

@benhillis
Ben Hillis (benhillis) marked this pull request as draft November 3, 2026 22:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants