Skip to content

feat(ci): typecheck lexicon tests, scripts and examples, against a shrinking baseline (#1388) - #1392

Merged
lex00 merged 2 commits into
mainfrom
feat/typecheck-tests
Aug 3, 2026
Merged

feat(ci): typecheck lexicon tests, scripts and examples, against a shrinking baseline (#1388)#1392
lex00 merged 2 commits into
mainfrom
feat/typecheck-tests

Conversation

@lex00

@lex00 lex00 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Part of #1388 — the gating half. Working the backlog down is the other half.

The gap

CI typechecks packages/core and packages/k8s-client, and check-lexicons builds each lexicon's tsconfig.build.json — which excludes **/*.test.ts, correctly, since a published lexicon should not ship its tests. Nothing covered what fell between:

$ tsc -p lexicons/forgejo/tsconfig.build.json --listFiles | grep -c '\.test\.ts'
0
$ tsc -p packages/core/tsconfig.json     --listFiles | grep -c '\.test\.ts'
266

A type-level assertion written in a lexicon test was evaluated by nothing. That is why #1344's ChantConfig augmentation proofs had to live in compiled source instead of the test where they belong — and the next person would reach for the test file first.

Why a ratchet rather than switching it on

I filed #1388 estimating this at 56 example errors. Measured properly it is ten times that:

Population Error lines Covered today
packages/** (src and tests) 0 yes
lexicons/*/src non-test 0 yes
lexicons/*/src tests 405 no
lexicons/*/scripts 7 no
examples 546 no
total 958 across 118 files

A check that cannot pass is a check nobody runs. So this follows KNOWN_FAILURES in check-lexicons.ts: everything failing today is listed, and the list can only get smaller.

  • a file not in the baseline that fails → error, reported by name and line
  • a file in the baseline that now passes → error, with instructions to delete its line

Verified both directions:

$ npx tsx scripts/typecheck.ts        # after adding a broken test
1 file(s) newly failing the repo-wide typecheck:
  lexicons/forgejo/src/__ratchet-probe.test.ts
    (1,7): error TS2322: Type 'string' is not assignable to type 'number'.

$ npx tsx scripts/typecheck.ts        # after baselining a file that passes
1 baselined file(s) now typecheck cleanly:
  lexicons/forgejo/src/config.ts
  Remove them from scripts/typecheck-baseline.json …

Note on moduleResolution

bundler in this config, node everywhere else. Deliberate: tests and examples run under vitest and tsx, which resolve like a bundler. Shipped code keeps its node-resolution build configs, so nothing about the published packages' resolution story changes. Under node the same run reports 763 lines, a large share of them resolution artifacts rather than real type errors — which would make the baseline mostly noise.

For whoever takes the backlog

Concentrated by lexicon: github 118 lines, gcp 105, temporal 81, aws 51, k8s 37. One lexicon at a time drops the baseline in reviewable chunks.

Verification

  • npm run typecheck118 known failure(s), no regressions
  • New CI step in the check job, after the lexicon completeness contract

🤖 Generated with Claude Code

https://claude.ai/code/session_01Jp6SP7ZiavhoEB1AFuwwoJ

lex00 and others added 2 commits August 2, 2026 22:28
…rinking baseline (#1388)

CI typechecks packages/core and packages/k8s-client, and check-lexicons builds
each lexicon's tsconfig.build.json — which excludes `**/*.test.ts`, correctly,
since a published lexicon should not ship its tests. Nothing covered what fell
between: `tsc -p lexicons/forgejo/tsconfig.build.json --listFiles | grep -c
'.test.ts'` returns 0, while packages/core typechecks its 266.

So a type-level assertion written in a lexicon test was evaluated by nothing.
That is why #1344's `ChantConfig` augmentation proofs had to live in compiled
source rather than in the test where they belong, and the next person would
reach for the test file first.

Turning the coverage on outright is not available. Measured across the
uncovered populations, under bundler resolution (how vitest and tsx actually
resolve):

  lexicons/*/src tests        405 error lines
  lexicons/*/scripts            7
  examples                    546
  total                       958   across 118 files

A check that cannot pass is a check nobody runs, so this ratchets instead —
the same shape as KNOWN_FAILURES in check-lexicons.ts. Every file failing today
is baselined; a file not in the baseline failing is an error, and a baselined
file that now passes is also an error, with instructions to delete its line.
The backlog can only shrink, and a test written tomorrow is typechecked from
the moment it exists.

Verified in both directions: a deliberately broken new test file is reported by
name and line, and a baselined file that starts passing fails with "remove them
from the baseline".

moduleResolution is bundler in this config and node everywhere else. Tests and
examples run under vitest and tsx; shipped code keeps its node-resolution build
configs, so nothing about the published packages changes. Under node the same
run reports 763 lines, many of them resolution artifacts rather than real
errors.

Working the 118 down is the other half of #1388.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jp6SP7ZiavhoEB1AFuwwoJ
…artifacts

The first baseline was captured in a clone with stale `src/generated/`, so it
recorded 33 files that fail only there. CI runs every lexicon's prepack before
typechecking, so those files pass — and the ratchet caught it in the direction
it was built to catch: "33 baselined files now typecheck cleanly, remove them".

Baseline drops 118 → 85. The script now says so when writing one, and the
module doc records the trap.
@lex00
lex00 force-pushed the feat/typecheck-tests branch from 357999a to 71bf6ac Compare August 3, 2026 04:33
@lex00
lex00 merged commit e53b0a1 into main Aug 3, 2026
5 checks passed
@lex00
lex00 deleted the feat/typecheck-tests branch August 3, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant