Repository navigation
fix: preserve version-named leaf pages during discovery - #136
Merged
Merged
Conversation
Only consider non-terminal path segments as version prefixes during deduplication. Preserve release notes, changelog entries, and version roots without changing child-page version selection. Add regression coverage for trailing slashes, sibling leaves, preferred versions, and both llms.txt and sitemap discovery. Fixes #135.
Document the rationale for preserving version-named leaf pages, the separate base-URL version semantics, sampling and request-budget tradeoffs, and the existing deduplication threshold interaction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #135.
Only consider non-terminal path segments as version prefixes in
deduplicateVersionedUrls(). A terminal version-like segment remains part of the page's grouping key, preserving release notes, changelog entries, and version roots.Independent Assessment
The reported collision is reproducible:
/docs/v1.0/groups with/docs/, and the unversioned parent wins. Sibling release pages also collapse to one even without a section index, so giving version leaves a separate shared namespace would not solve the data loss.Restricting prefix detection to segments with a following page path is the smallest conservative fix. Ordinary versioned child pages still deduplicate, including explicit preferred-version selection.
extractVersionFromUrl()is intentionally unchanged because a base URL ending in/v1/must still select v1 child pages.Tradeoff: version roots such as
/docs/v1/and/docs/v2/remain separate discovery candidates, even with an explicit version preference. URL structure alone cannot reliably distinguish these roots from distinct version-named release documents. This can increase the discovery pool; normal sampling limits still apply.This change addresses version deduplication only; the separate file-type filtering issue in #134 is outside its scope.
Regression Coverage
llms.txtand sitemap paths.Before the fix, all four initial regression cases failed while the 13 existing deduplication tests passed. After the fix, all 161 page-discovery tests pass, including eight new regression cases.
Validation
npm test -- --reporter=dot: 1,741 tests passed across 67 files.npm run lint: passed.npm run build: passed.npm run version:check: passed.npm run format:check: passed.After applying Prettier to the changed test file, reran all 161 page-discovery tests and the repository formatting check successfully.