Test the package against every supported .NET SDK and FSharp.Core - #12
Merged
Conversation
The Expecto suite uses a project reference, so it only ever exercises the FSharp.Core the repository itself resolves. This adds a second suite that consumes the packed nupkg from inside containers with pinned SDK and FSharp.Core versions, which is what a real consumer's toolchain looks like. Each case formats a value through the library and through the FSharp.Core of that combination, so FSharp.Core is the oracle and any divergence is marked MISMATCH; golden files pin the output so silent drift also shows up. One image is built per combination and the program variant is chosen at run time. Run it with `./build.sh DockerCompat`. Only the combination the repository builds against fails the build; the rest are reported in artifacts/CompatReport.md. The matrix shows SDK 3.1, 5.0 and 6.0 cannot consume the package at all: built by the .NET 10 SDK, it carries its F# metadata only in the compressed form F# 8 introduced, so older compilers see no F# metadata in the assembly. That puts the real consumer floor at SDK 8.0 rather than at the declared FSharp.Core 4.5.0. Those rows are left failing and tolerated so the boundary stays measured. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Since F# 8 the compiler compresses its metadata resource by default, naming it FSharpSignatureCompressedData. Compilers older than that only look for the uncompressed FSharpSignatureData, find nothing, and conclude the assembly has no F# metadata at all — so consumers on SDK 3.1, 5.0 and 6.0 could not see the namespace even though NuGet resolved and passed them the assembly. Emitting the uncompressed resource fixes every failing row of the compatibility matrix: it now passes from SDK 3.1 with FSharp.Core 4.5.0 through SDK 10 with FSharp.Core 10.1.302, with byte-identical output across all of them. All rows are promoted to mandatory so any regression fails the build. Shipping several DLLs would not have worked instead: NuGet selects by the consumer's target framework rather than their compiler version, and every target framework in one build is compiled by the same fsc anyway. Costs about 29 KB of assembly size (84 KB to 113 KB). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
It existed so combinations that could not work yet would be reported without blocking, but every combination passes now, so the flag was uniformly true and the two failure categories only added noise to the report and the summary. Any failure now fails the build. A combination that stops working should be fixed or removed rather than quietly downgraded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
They were string constants, so nothing stopped a typo or an arbitrary string from reaching the matrix, and the compiler could not tell that the three variants are the whole set. Variant.name converts to the string the file names and entrypoint.sh use, so that spelling now lives in exactly one place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The build context is the repository root because the image needs both compat-tests/ and the packed nupkg, which forced the ignore file to the repository root where it was the only Docker-related file outside compat-tests/. Docker prefers an ignore file named after its Dockerfile and sitting beside it over a .dockerignore at the context root, so it can live in compat-tests/. The patterns are unchanged: they are relative to the context root either way. Verified by copying the whole context into a throwaway image: 228 KB with the ignore file against 95 MB without it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Markdown report and its CI artifact were an extra hop for information a CI log shows directly. The summary now carries the SDK and FSharp.Core versions of each row, and a failure prints the captured compiler output or the differing lines, which is what the report existed to preserve. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a compatibility test suite that installs the built
.nupkginto small consumer programs inside Docker containers with pinned .NET SDK and FSharp.Core versions, and compares their output against golden files.The existing Expecto suite uses a
ProjectReference, so it only ever exercises whatever FSharp.Core the repository itself resolves. This covers the other axis: what a real consumer's toolchain produces.Each case formats a value twice — once through the library, once through the FSharp.Core of that combination — so FSharp.Core is the oracle and any divergence is marked
MISMATCH. Golden files then pin the output so silent drift also shows up.Three program variants, each a superset of the last:
Floor45(every specifier, flag, width/precision and star form),Interpolation50(string interpolation),Binary60(%B). A combination only runs the variants its era supports.The bug it found
The package could not be consumed at all by SDK 3.1, 5.0 or 6.0. NuGet resolved it and passed
-r:BlackFox.MasterOfFoo.dllto the compiler, but compilation failed withThe namespace or module 'BlackFox' is not defined.Since F# 8 the compiler compresses its metadata resource by default, naming it
FSharpSignatureCompressedData. Older compilers only look for the uncompressedFSharpSignatureData, find nothing, and conclude the assembly contains no F# metadata. Building with--compressmetadata-fixes every failing row, at a cost of ~29 KB of assembly size (84 KB → 113 KB).Notably, shipping multiple DLLs would not have helped: NuGet selects by the consumer's target framework, not their compiler version, and every target framework in one build is compiled by the same
fscanyway.Result
All 13 checks pass, with byte-identical output across FSharp.Core 4.5.0 through 10.1.302 — no behavioural drift at all in ~70 format cases.
Notes
./build.sh DockerCompat; writesartifacts/CompatReport.md.CI, since it builds several images.dockercall pinslinux/amd64because the old SDK images are only published for that architecture.linuxandwindowsjobs are untouched.🤖 Generated with Claude Code