feat(ci): typecheck lexicon tests, scripts and examples, against a shrinking baseline (#1388) - #1392
Merged
Conversation
…rinking baseline (#1388) CI typechecks packages/core and packages/k8s-client, and check-lexicons builds each lexicon's tsconfig.build.json — which excludes `**/*.test.ts`, correctly, since a published lexicon should not ship its tests. Nothing covered what fell between: `tsc -p lexicons/forgejo/tsconfig.build.json --listFiles | grep -c '.test.ts'` returns 0, while packages/core typechecks its 266. So a type-level assertion written in a lexicon test was evaluated by nothing. That is why #1344's `ChantConfig` augmentation proofs had to live in compiled source rather than in the test where they belong, and the next person would reach for the test file first. Turning the coverage on outright is not available. Measured across the uncovered populations, under bundler resolution (how vitest and tsx actually resolve): lexicons/*/src tests 405 error lines lexicons/*/scripts 7 examples 546 total 958 across 118 files A check that cannot pass is a check nobody runs, so this ratchets instead — the same shape as KNOWN_FAILURES in check-lexicons.ts. Every file failing today is baselined; a file not in the baseline failing is an error, and a baselined file that now passes is also an error, with instructions to delete its line. The backlog can only shrink, and a test written tomorrow is typechecked from the moment it exists. Verified in both directions: a deliberately broken new test file is reported by name and line, and a baselined file that starts passing fails with "remove them from the baseline". moduleResolution is bundler in this config and node everywhere else. Tests and examples run under vitest and tsx; shipped code keeps its node-resolution build configs, so nothing about the published packages changes. Under node the same run reports 763 lines, many of them resolution artifacts rather than real errors. Working the 118 down is the other half of #1388. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jp6SP7ZiavhoEB1AFuwwoJ
…artifacts The first baseline was captured in a clone with stale `src/generated/`, so it recorded 33 files that fail only there. CI runs every lexicon's prepack before typechecking, so those files pass — and the ratchet caught it in the direction it was built to catch: "33 baselined files now typecheck cleanly, remove them". Baseline drops 118 → 85. The script now says so when writing one, and the module doc records the trap.
lex00
force-pushed
the
feat/typecheck-tests
branch
from
August 3, 2026 04:33
357999a to
71bf6ac
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1388 — the gating half. Working the backlog down is the other half.
The gap
CI typechecks
packages/coreandpackages/k8s-client, andcheck-lexiconsbuilds each lexicon'stsconfig.build.json— which excludes**/*.test.ts, correctly, since a published lexicon should not ship its tests. Nothing covered what fell between:A type-level assertion written in a lexicon test was evaluated by nothing. That is why #1344's
ChantConfigaugmentation proofs had to live in compiled source instead of the test where they belong — and the next person would reach for the test file first.Why a ratchet rather than switching it on
I filed #1388 estimating this at 56 example errors. Measured properly it is ten times that:
packages/**(src and tests)lexicons/*/srcnon-testlexicons/*/srctestslexicons/*/scriptsA check that cannot pass is a check nobody runs. So this follows
KNOWN_FAILURESincheck-lexicons.ts: everything failing today is listed, and the list can only get smaller.Verified both directions:
Note on
moduleResolutionbundlerin this config,nodeeverywhere else. Deliberate: tests and examples run under vitest and tsx, which resolve like a bundler. Shipped code keeps itsnode-resolution build configs, so nothing about the published packages' resolution story changes. Undernodethe same run reports 763 lines, a large share of them resolution artifacts rather than real type errors — which would make the baseline mostly noise.For whoever takes the backlog
Concentrated by lexicon: github 118 lines, gcp 105, temporal 81, aws 51, k8s 37. One lexicon at a time drops the baseline in reviewable chunks.
Verification
npm run typecheck→118 known failure(s), no regressionscheckjob, after the lexicon completeness contract🤖 Generated with Claude Code
https://claude.ai/code/session_01Jp6SP7ZiavhoEB1AFuwwoJ