docs: name the prior art on token measurement, and state the estimator's real error - #119
Open
singhharsh1708 wants to merge 3 commits into
Open
docs: name the prior art on token measurement, and state the estimator's real error#119singhharsh1708 wants to merge 3 commits into
singhharsh1708 wants to merge 3 commits into
Conversation
…r's error The README claimed no other tool surfaces a skill's token cost. That is no longer true: skills-check (npx skills-check budget) counts tokens per skill with a real cl100k_base tokenizer, breaks them down per section, detects redundancy between skills, and fails a build over a ceiling. Leaving the claim up would be the same credibility problem the 0.11.0 pass existed to fix, on the one number this project is known for. Corrected to what is still specific here: the per-target half. The same skill has a different standing cost on every agent because each loads it differently, and that falls out of a compile step that already knows each target's loading mode — skills-check measures the file, not the pair. Also states the estimator's real error rather than "a few percent". Measured against o200k_base: length/4 runs 2-11% high on this corpus (prereview 567 vs 515, stub 40 vs 35), biased toward failing budgets early, and the 14x ratio is 14.7x under a real tokenizer.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
documentation
…ng one The previous wording said length/4 runs 'about 2-11% high' and called that the safe direction for budget enforcement. Measuring it against o200k_base over the artifacts the benchmark emits, neither half holds: prereview stub 40 est 35 real 1.14x prereview body 567 est 515 real 1.10x review-checklist stub 19 est 14 real 1.36x review-checklist body 880 est 942 real 0.93x The range is 7% low to 36% high, not 2-11% high, and the direction flips with content: prose over-counts, repetitive numbered lists under-count. A skill shaped like review-checklist can therefore pass a budget it actually exceeds, which is the opposite of the claimed bias, and --strict will not catch it. The ratio is more robust but not exact either: prereview's 14x is 14.7x, and review-checklist's 47x is 68x — the estimator understates that gap by a third, so 'within half a multiple' was wrong too. Says how the comparison was taken and that it is a one-off, since kitbash ships no tokenizer and cannot regenerate it.
…ring it The error bar the benchmark quotes was a number someone measured once and typed in. It was wrong — 'about 2-11% high' against a real range of -7% to +36% — and nothing in the repo could have caught that, which is the same class of problem as a hand-maintained copy of the loading-mode table. Adds scripts/tokenizer-check.mjs (npm run tokencheck): measures the artifacts the benchmark emits against o200k_base and prints the range and both standing-tax ratios, so the figures in docs/benchmarks/README.md are regenerable. gpt-tokenizer is a devDependency and runs in no shipped code path; runtime dependency count is still zero. Cross-checked against tiktoken on four artifacts — identical counts. Extracts the corpus into scripts/bench-fixtures.mjs so the table and its error bar describe the same bytes rather than two copies of the fixture setup. benchmark.mjs regenerates docs/benchmarks/README.md byte-identically across the refactor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Retracts a claim the README should not have been making, and replaces an error bar that was measured on the wrong corpus.
skills-checkcounts tokens per skill with a realcl100k_basetokenizer, breaks them down per section, and fails a build over a ceiling, so "I have not found another tool that surfaces that number" is no longer true. What survives is the per-target half: the same skill has a different standing cost on every agent because each one loads it differently, and that falls out of a compile step which already knows each target's loading mode.skills-checkmeasures the file; Kitbash measures the pair.The estimator's stated error, "about 2-11% high", was measured over
SKILL.md,README.md,spec/SPEC.mdanddocs/roadmap.md, then attached to the benchmark table, whose numbers come from entirely different bytes. Measured over the artifacts the benchmark actually emits,length / 4runs 7% low to 36% high, and the direction flips with content: prose over-counts, while the repetitivereview-checklistbody under-counts by 7%. So the claim that the bias runs in the safe direction for budget enforcement was wrong, and a skill shaped like that one can pass a budget it actually exceeds with--strictnone the wiser. The ratio holds up better but is not exact either:prereview's 14x is 14.7x under a real tokenizer, andreview-checklist's 47x is 68x, so "within half a multiple" was wrong too.npm run tokencheckregenerates all of it from the same artifacts the benchmark measures, so the next figure that drifts fails visibly instead of sitting in a paragraph.gpt-tokenizeris a devDependency and runs in no shipped code path, leaving the runtime dependency count at zero; its counts were cross-checked againsttiktokenon four artifacts and matched exactly.The corpus moved into
scripts/bench-fixtures.mjsso the table and its error bar cannot describe different bytes again, which is the mistake that produced the wrong number in the first place.benchmark.mjsregeneratesdocs/benchmarks/README.mdbyte-identically across that refactor.Suite green on macOS, Ubuntu and Windows. Merge after #118 to avoid a changelog collision.