docs(testing): correct four stale claims in TESTING.md - #1404
Merged
Conversation
**Test Distribution** claimed 19 core / 11 aws / 7 cli test files. Actual: 219, 102, 49. The same failure as the lexicons' At-a-Glance tables — a count written into prose, frozen at whenever it was written, checked by nothing. Replaced with a description of where tests live plus the `find` that counts them, rather than a number that will be wrong again in a month. **`packages/aws/src/spec/fetch.test.ts`** does not exist and has not since the lexicons moved out of `packages/`. The file is at `lexicons/aws/...`. Two occurrences, one of them a command to run. **"all 6 lexicons"**, five occurrences, was right in one place and wrong in four. `integration.sh` really does test six (aws, azure, gcp, gitlab, k8s, docker — six `test_lexicon` calls); `npm-smoke.sh` and `smoke.sh` both iterate eight, adding fly and fountain. Each claim now names its own script's coverage instead of sharing one number between harnesses that differ. Found by checking TESTING.md's falsifiable claims against source: every `just` recipe against the justfile, every `npx tsx` path against the tree, every backticked file path, and every count. The recipes and script paths were all correct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jp6SP7ZiavhoEB1AFuwwoJ
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second slice of the docs audit. Same method as #1399: check the falsifiable claims against source rather than read for plausibility.
Test Distribution was frozen at an early state
The same failure as the lexicons' At-a-Glance tables in #1312 — a count written into prose, correct once, checked by nothing since. Replaced with a description of where tests live plus the
findthat counts them, rather than three numbers that will be wrong again in a month.A path that has not existed since the reorganization
packages/aws/src/spec/fetch.test.ts, twice — once as prose, once as a command to run. There is nopackages/aws; the file is atlexicons/aws/src/spec/fetch.test.ts."all 6 lexicons" — right once, wrong four times
The number was shared between harnesses that genuinely differ:
test/integration.shtest_lexiconcalls)test/npm-smoke.shtest/smoke.shSo the claim was accurate for
integration.shand understated the other two by two lexicons each. Each claim now names its own script's coverage.What checked out
Every
justrecipe named in the file exists in the justfile; everynpx tsxscript path exists; the remaining backticked paths are relative filenames within a discussed directory rather than repo paths.🤖 Generated with Claude Code
https://claude.ai/code/session_01Jp6SP7ZiavhoEB1AFuwwoJ