diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 24345cf..2498f06 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -81,6 +81,7 @@ export default defineConfig({ { text: 'Run Locally', link: '/run-locally' }, { text: 'Improve Your Score', link: '/improve-your-score' }, { text: 'CI Integration', link: '/ci-integration' }, + { text: 'Documentation at Scale', link: '/documentation-at-scale' }, ], }, { diff --git a/docs/ci-integration.md b/docs/ci-integration.md index 5d7253d..36a7d6b 100644 --- a/docs/ci-integration.md +++ b/docs/ci-integration.md @@ -50,6 +50,8 @@ Checks that fail cause the test to fail. Checks that warn still pass (they're in ## Running a subset of checks +For products on a shared documentation host, [Documentation at Scale](/documentation-at-scale) shows how to run repeatable checks on pages your team chooses, both before deployment and on a schedule. It includes a curated config and explains what those checks do and do not establish about agents' ability to find your documentation. + To exclude specific checks, use `skipChecks`. For example, a local preview server may not implement the content negotiation or cache headers supplied by your production server: ```yaml diff --git a/docs/documentation-at-scale.md b/docs/documentation-at-scale.md new file mode 100644 index 0000000..d91894c --- /dev/null +++ b/docs/documentation-at-scale.md @@ -0,0 +1,221 @@ +# Documentation at Scale + +Use this guide as a rule of thumb when your site has roughly 500 or more +documentation pages, or two or more products sharing a host that need +separate results. These are workflow guidelines, not crawl limits. Smaller +sites can benefit too when their content types, locales, or versions need +separate attention. + +The default run samples at most 50 pages. As the corpus grows or becomes more +varied, which pages are selected matters more than the total count alone. +Start with the product and audience you own, maintain a representative page +set, and understand what your checks do and do not cover. + +AFDocs samples documentation; it is not an exhaustive crawler. A completed +run tells you about the pages and checks it evaluated, not every page on the +host. + +## Define the scope your team owns + +Use a separate config for each product, locale, or supported version that +needs its own results and owner. Choose a base URL whose path actually +contains that content. + +For example, a Microsoft Learn team might own content under +`https://learn.microsoft.com/en-us/dotnet/`. Neither the whole Learn host nor +the general `/en-us/docs/` landing page is a useful substitute for that +product scope. In the live run that motivated this guidance, `/en-us/docs/` +matched none of the examined sitemap URLs; AFDocs finished by checking the +base page instead. + +With `https://learn.microsoft.com/en-us/dotnet/` as your starting URL, AFDocs +looks for pages whose addresses begin with that URL. This shared beginning +is the **[product prefix](/glossary#product-prefix)**. Changing the starting URL to +`https://learn.microsoft.com/` opens the search to other products and +languages, including Azure and SQL Server. A sample from that wider search +would no longer tell you specifically how your .NET documentation performs. + +On a large site, the published list of pages may be split across many files. +AFDocs limits how much of that list it reads, so it may stop before reaching +your product even when your starting URL is correct. If that happens, keep +the product URL and supply your own page list for page checks, as shown +below. Changing to the whole-site URL just to remove the warning does not +solve the problem of finding a useful sample for your team. + +A **[curated page list](/glossary#curated-page-list)** is a list of pages you +choose yourself. For an assessment of English-language .NET documentation: + +- Include `https://learn.microsoft.com/en-us/dotnet/core/install/`, an installation guide. +- Include `https://learn.microsoft.com/en-us/dotnet/csharp/`, the C# documentation. +- Leave out `https://learn.microsoft.com/en-us/azure/virtual-machines/overview`, which belongs to a different product. +- Leave out `https://learn.microsoft.com/de-de/dotnet/core/install/`, which is a German-language version, unless you intend to assess that language too. + +When you use a curated page list, AFDocs scores the pages you list, even if +they belong to another product or language. Setting the starting URL to the +English .NET section does not remove the Azure or German pages from your list. +Review the list to make sure it contains the pages your team wants results for. + +## Maintain a representative page set + +Commit a curated config alongside your documentation. Include examples of the +content and templates agents need to consume: + +- An installation page and a first successful task. +- A tutorial with commands and code examples. +- An API or command reference with tables and parameters. +- A page with tabs, generated content, or other complex rendering. +- An upgrade or troubleshooting page used by existing customers. + +These are selection criteria, not a statistical sampling guarantee. Add pages +when a new template ships or a regression reveals a missing case. Retire +obsolete entries deliberately; do not replace a failing page just to improve +the score. Tags can group results by content type or ownership, but AFDocs +does not use them to balance the sample. + +### Routine page checks + +The following example uses placeholder URLs for one product. Replace them +with real pages your team owns and save the config as something like +`agent-docs.YOUR_PRODUCT-pages.yml`. For example, you might name the +config below `agent-docs.widgets-pages.yml` and include these specific URLs: + +```yaml +url: https://docs.example.com/en-us/widgets/ + +options: + samplingStrategy: curated + requestDelay: 500 + maxConcurrency: 1 + +checks: + - markdown-url-support + - content-negotiation + - page-size-html + - page-size-transfer + - auth-gate-detection + +pages: + - url: https://docs.example.com/en-us/widgets/install + tag: getting-started + - url: https://docs.example.com/en-us/widgets/tutorials/first-app + tag: getting-started + - url: https://docs.example.com/en-us/widgets/reference/commands + tag: reference + - url: https://docs.example.com/en-us/widgets/guides/configuration + tag: guides + - url: https://docs.example.com/en-us/widgets/troubleshooting + tag: guides +``` + +Run it with: + +```bash +npx afdocs check --config agent-docs.widgets-pages.yml --format scorecard +``` + +This deliberately limited check set evaluates Markdown access, HTML size, +and authentication behavior without walking a sitemap or the site's llms.txt +links. It is a starting point, not a full assessment of agent-friendliness. +Expand it using the [Checks Reference](/checks/), including dependencies where +required. An explicit `checks` list does not pick up newly added checks +automatically; review it when upgrading AFDocs. + +All curated entries are used for the page sample; `maxLinksToTest` does not +truncate the list. If you provide 80 URLs and set `maxLinksToTest` to 50, +AFDocs checks all 80 URLs you provide. Individual checks still apply their +normal eligibility rules. Five listed pages also does not mean five requests: +Markdown URL candidates, content negotiation, redirects, and other checks +can require additional requests. + +Adding other checks can change the request scope. In particular, +`llms-txt-coverage` reads sitemaps independently of the curated sample, and +llms.txt link checks follow links published in that file. Curated sampling +does not restrict every check to fetching only the listed URLs. + +For details on tags and config selection, see the +[Config File Reference](/reference/config-file#pages-optional). + +## What curated checks don't tell you + +A maintained page list answers whether known pages remain usable. It does +not establish whether an agent can discover new content or whether llms.txt +covers the product's documentation. + +Automatic discovery can work when AFDocs reaches the relevant published +lists of pages. On a large shared site, it can stop before finding your +product. In our live test of Microsoft Learn's .NET section, AFDocs reached +its sitemap-reading limit while reading lists of archived content. It found +no .NET pages and checked only the landing page. Coverage was skipped +because no llms.txt was found at the locations AFDocs checked. + +For a site in that situation, use the curated config above for repeatable +page checks. A quick run or a good score does not establish that agents can +find your product's pages. AFDocs does not yet let you choose a product's +sitemap in a config file; that work is tracked in +[#142](https://github.com/agent-ecosystem/afdocs/issues/142). Until it can +reach the relevant page lists, it cannot provide a reliable product-wide +discovery or coverage assessment for that site. + +On a shared platform, product teams can maintain page samples and fix their +content, while platform owners handle shared rendering, Markdown delivery, +index publication, and server behavior. Send findings to the team that can +change the behavior rather than treating every failure as a writer's task. + +## Recognize incomplete evidence + +Review the actual tested pages and check results before interpreting a +score. Use `--verbose` with your curated config for investigation, or retain +a JSON report for CI: + +```bash +npx afdocs check --config agent-docs.widgets-pages.yml --format json --score > widgets-pages-report.json +``` + +If you also run checks that find pages automatically or read sitemaps, review +their discovery warnings. The following findings limit what you can conclude: + +| Finding | What to do | +| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| A path prefix matches few or no examined URLs | Check the prefix against actual content URLs. If it is correct, a shared index may not have reached your product; use a curated page set for page checks and investigate index scope separately. | +| A sitemap budget stops discovery | Treat the discovered set and coverage result as partial. Review the reported source with the platform owner. Do not assume increasing `--max-links` raises the sitemap budget. | +| Only the base page was tested | Do not interpret that as a product-wide assessment. Check discovery warnings, then provide a representative list or repair discovery. | +| A check is skipped | Inspect its dependencies and applicability. A skipped coverage check does not establish coverage. | +| Requests are blocked, rate-limited, or time out | Review the request summary and network context; coordinate the test cadence and CI environment with the platform owner. | + +The [CLI Reference](/reference/cli#url-discovery) describes sitemap budgets +and their limitations. Limits apply per walk, not per scan; the byte budget +is checked between responses. They prevent a runaway walk but do not certify +that enough of the corpus was discovered. + +Warnings do not normally cause a failing CLI exit code. A successful command +is therefore not proof of complete discovery. Preserve warnings and check +details with the report, especially when deciding whether a result is +suitable as a release gate. + +## Make results useful in CI + +Run curated regression checks against a preview of the change on pull +requests. Check that the configured page URLs target that preview, not the +production site. Run the same curated page list against the deployed site on +release or on a schedule to evaluate the server behavior a local preview +cannot reproduce. See +[Run Locally](/run-locally) and +[CI Integration](/ci-integration#checking-a-build-and-the-deployed-site). + +Pin the AFDocs version in your project's dependencies, set a CI job timeout, +and stagger product jobs that share a host. Request delay and concurrency +settings apply to each run; ten concurrent jobs do not share one limiter. +Lowering `--max-links` reduces a discovered page sample, not all discovery +traffic, and does not reduce an explicit curated list. + +Keep the scope, page list, tags, selected checks, AFDocs version, and target +environment with each report. Compare like-for-like runs. A score for five +curated pages and a selected check set is not directly comparable to a +full-check score for another product or a different page sample. Do not average +product scores into a host-wide score without defining what that aggregate +would represent. + +For a team on a platform such as Microsoft Learn, the first useful milestone +is a repeatable product-owned regression run with actionable findings. A +platform-wide assessment needs a separately designed sampling and discovery +strategy, not a larger value of `--max-links`. diff --git a/docs/generate_llms_txt b/docs/generate_llms_txt index 11dee9a..c625467 100755 --- a/docs/generate_llms_txt +++ b/docs/generate_llms_txt @@ -32,6 +32,7 @@ echo "- [Quick Start](${BASE_URL}/quick-start.md)" >> "$OUTPUT" echo "- [Run Locally](${BASE_URL}/run-locally.md)" >> "$OUTPUT" echo "- [Improve Your Score](${BASE_URL}/improve-your-score.md)" >> "$OUTPUT" echo "- [CI Integration](${BASE_URL}/ci-integration.md)" >> "$OUTPUT" +echo "- [Documentation at Scale](${BASE_URL}/documentation-at-scale.md): product-owned page samples, discovery limits, and CI on shared hosts" >> "$OUTPUT" cat >> "$OUTPUT" << 'SECTION' diff --git a/docs/glossary.md b/docs/glossary.md index cc0c825..9f78078 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -14,6 +14,10 @@ A page a bot-protection system shows instead of the content, asking the visitor On a different website (a different scheme, host, or port) from the one being tested. AFDocs never fetches cross-origin links to grade a site, because another site's availability is not this site's result. +## Curated page list + +A list of pages you choose for AFDocs to check, such as an installation guide, a tutorial, and an API reference. Supply their URLs through `pages` in a config file or the `--urls` CLI option. AFDocs uses your list instead of automatically choosing the page sample. Checks that read indexes or follow links can still request other pages. See [choosing representative pages](/documentation-at-scale#maintain-a-representative-page-set) for an example. + ## Hydration payload Data that a JavaScript framework embeds inside a page, usually in script tags, so the interactive version of the page can be rebuilt in the browser. It often contains a full copy of the page's content plus menus and metadata. Readers never see it; agents download all of it. See [page-size-transfer](/checks/page-size#page-size-transfer). @@ -22,6 +26,10 @@ Data that a JavaScript framework embeds inside a page, usually in script tags, s The two ways an agent can get a page. On the HTML path it fetches the same page a browser would and converts it to text. On the markdown path it fetches a markdown version, either by appending `.md` to the URL, by asking for markdown ([content negotiation](/checks/markdown-availability#content-negotiation)), or by following links from `llms.txt`. The markdown path is cheaper and cleaner, but only helps agents that find it. +## Product prefix + +The shared beginning of page addresses for one product. For example, `https://learn.microsoft.com/en-us/dotnet/` identifies the English-language .NET section, rather than the whole Microsoft Learn site. AFDocs uses this part of the starting URL to narrow the pages it finds during discovery. It does not remove pages from a curated list or restrict every request to that section. See [defining your team's scope](/documentation-at-scale#define-the-scope-your-team-owns). + ## RAG Retrieval-augmented generation. A pipeline that cuts fetched content into chunks, stores them, and later pulls back only the chunks that seem relevant to a question. A note or a link that lands in one chunk loses the context of the content it was about. diff --git a/docs/public/llms.txt b/docs/public/llms.txt index 73d56ee..87fd6f8 100644 --- a/docs/public/llms.txt +++ b/docs/public/llms.txt @@ -16,6 +16,7 @@ - [Run Locally](https://afdocs.dev/run-locally.md) - [Improve Your Score](https://afdocs.dev/improve-your-score.md) - [CI Integration](https://afdocs.dev/ci-integration.md) +- [Documentation at Scale](https://afdocs.dev/documentation-at-scale.md): product-owned page samples, discovery limits, and CI on shared hosts ## Checks Reference diff --git a/docs/quick-start.md b/docs/quick-start.md index c8d5fed..e429096 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -83,7 +83,7 @@ See the [Checks Reference](/checks/) for the full list of check IDs and dependen ## Check a specific page -Skip page discovery entirely and check just one URL with `--sampling none`: +Skip page-sample discovery and use just one URL for page-level checks with `--sampling none`: ```bash npx afdocs check https://docs.example.com/api/auth --sampling none @@ -103,7 +103,7 @@ If you want to check a handful of pages without running full discovery, pass the npx afdocs check https://docs.example.com --urls https://docs.example.com/quickstart,https://docs.example.com/api/auth ``` -This skips page discovery and runs all checks against exactly those URLs. You can tag pages for grouped scoring by defining them in a config file: +This skips page-sample discovery. Page-level checks use the listed URLs; index, link, and coverage checks can still fetch other resources. You can tag pages for grouped scoring by defining them in a config file: ```yaml # agent-docs.config.yml @@ -125,6 +125,8 @@ npx afdocs check --format scorecard The scorecard will include a Tag Scores section showing how each group of pages scores, with a per-check breakdown of what's passing and failing within each tag. The JSON output (`--format json --score`) includes full per-page detail for each tag. See [Config File Reference](/reference/config-file) for the full `pages` schema. +As a rule of thumb, if your site has roughly 500 or more documentation pages, or two or more products sharing a host that need separate results, see [Documentation at Scale](/documentation-at-scale) for representative page sets, repeatable page checks, and the limits of automatic discovery. These are workflow guidelines, not crawl limits; smaller sites can benefit too when their content types, locales, or versions need separate attention. + ## Get consistent results between runs By default, AFDocs randomly samples pages, so results can vary between runs. For reproducible results (useful when verifying a fix), use deterministic sampling: diff --git a/docs/reference/cli.md b/docs/reference/cli.md index ea47430..494565d 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -96,8 +96,10 @@ Some checks depend on others. If you include a check without its dependency, the - **`random`**: Shuffle discovered URLs and take the first N. Fast and broad, but results vary between runs. Useful for spot-checking pages across a large corpus. - **`deterministic`**: Sort discovered URLs alphabetically and pick an even spread. Produces the same sample on repeated runs as long as the URL set is stable. Useful for CI or when verifying a fix. -- **`curated`**: Test a specific set of pages listed in the config file's `pages` field or passed via `--urls`. Skips discovery entirely. Useful for ongoing monitoring of representative pages or focused evaluation of specific sections. -- **`none`**: Skip discovery entirely. Only check the URL you pass on the command line. +- **`curated`**: Use the pages listed in the config file's `pages` field or passed via `--urls` for page-level checks, without page-sample discovery. Useful for ongoing monitoring of representative pages or focused evaluation of specific sections. +- **`none`**: Skip page-sample discovery and use only the URL you pass on the command line for page-level checks. + +Neither `curated` nor `none` disables a selected check's own index or link fetching. In particular, sitemap coverage runs independently of the page sample. `--max-links` does not truncate a curated list or cap total requests. See [Documentation at Scale](/documentation-at-scale) for choosing a predictable page-check workflow. ```bash # Reproducible results @@ -127,6 +129,19 @@ afdocs check https://docs.example.com --max-links 100 When `afdocs` discovers pages from a sitemap or `llms.txt`, it automatically filters out duplicate locale and version variants so you get a representative sample of unique content. +Sitemap walks stop after 20 sitemap fetch attempts or after completed sitemap +bodies total 50 MiB of decoded data, even when no URLs match the base path. +The byte limit is checked between responses, so the final body can exceed it. +These limits apply per walk, including coverage's docs-sitemap fallback, not +to the whole scan. When a limit prevents further fetching, discovery reports +partial results under `--verbose`; coverage includes `sitemapWarnings` in its +result details. + +A discovery warning also identifies a path prefix that matches fewer than 1% +of examined same-site URLs. Check for a typo or use a broader base URL, for +example `afdocs check https://learn.microsoft.com/en-us --verbose` instead of +scoping discovery to `/en-us/docs`. The warning does not change your scope. + The resolution order for both flags is: 1. **Explicit flag** (`--doc-locale`, `--doc-version`) if provided diff --git a/docs/reference/config-file.md b/docs/reference/config-file.md index 02bb9c0..de395bf 100644 --- a/docs/reference/config-file.md +++ b/docs/reference/config-file.md @@ -120,7 +120,7 @@ Override default runner options. All fields are optional: ### `pages` (optional) -A list of specific page URLs to test. When `pages` is present and no `samplingStrategy` is explicitly set, the strategy defaults to `curated`, which skips discovery and tests exactly the listed pages. +A list of specific page URLs to test. When `pages` is present and no `samplingStrategy` is explicitly set, the strategy defaults to `curated`, which skips page-sample discovery and uses the listed pages for page-level checks. Checks that read indexes, follow their links, or calculate sitemap coverage can still fetch other resources. Each entry can be a plain URL string or an object with `url` and an optional `tag` for grouped scoring: @@ -145,6 +145,8 @@ Tags are optional and can be mixed with plain URL strings. Pages without tags ar Note that `maxLinksToTest` does not apply to curated pages; all listed pages are tested. +Curated entries are not filtered by the base URL's path or by discovery's locale and version preferences. Keep the list within your intended scope. See [Documentation at Scale](/documentation-at-scale) for product-owned configs, repeatable page checks, and the limits of automatic discovery. + ## Config resolution The config loader searches for `agent-docs.config.yml` (or `.yaml`) starting from the current working directory and walking up the directory tree, similar to how ESLint and Prettier find their config files. This means the config works whether you're running the CLI from your project root or running a test file from a subdirectory. diff --git a/src/checks/observability/llms-txt-coverage.ts b/src/checks/observability/llms-txt-coverage.ts index 79e8d1d..4631d8d 100644 --- a/src/checks/observability/llms-txt-coverage.ts +++ b/src/checks/observability/llms-txt-coverage.ts @@ -2,7 +2,6 @@ import { registerCheck } from '../registry.js'; import { getUrlsFromCachedLlmsTxtWithOmitted, getUrlsFromSitemap, - parseSitemapUrls, } from '../../helpers/get-page-urls.js'; import { isSameSite } from '../../helpers/host-equivalence.js'; import { isNonPageUrl } from '../../helpers/to-md-urls.js'; @@ -259,38 +258,17 @@ const MAX_COVERAGE_SITEMAP_URLS = 50_000; * Many docs sites host their own sitemap that isn't referenced from robots.txt * (e.g., Loops /docs/sitemap.xml, Supabase /docs/sitemap.xml). */ -async function fetchDocsSitemap(ctx: CheckContext): Promise { +async function fetchDocsSitemap(ctx: CheckContext, warnings: string[]): Promise { const baseUrlPath = new URL(ctx.baseUrl).pathname.replace(/\/$/, ''); if (!baseUrlPath || baseUrlPath === '/') return []; const docsSitemapUrl = `${ctx.origin}${baseUrlPath}/sitemap.xml`; - try { - const response = await ctx.http.fetch(docsSitemapUrl); - if (!response.ok) return []; - const xml = await response.text(); - const parsed = parseSitemapUrls(xml); - - // If it's a sitemap index, follow one level - if (parsed.sitemapIndexUrls.length > 0) { - const urls: string[] = []; - for (const subUrl of parsed.sitemapIndexUrls) { - try { - const subResp = await ctx.http.fetch(subUrl); - if (!subResp.ok) continue; - const subXml = await subResp.text(); - const subParsed = parseSitemapUrls(subXml); - urls.push(...subParsed.urls); - } catch { - // Skip failed fetches - } - } - return urls; - } - - return parsed.urls; - } catch { - return []; - } + return getUrlsFromSitemap(ctx, warnings, { + sitemapUrls: [docsSitemapUrl], + maxUrls: MAX_COVERAGE_SITEMAP_URLS, + originOverride: ctx.effectiveOrigin ?? ctx.origin, + skipRefinement: true, + }); } /** @@ -358,7 +336,7 @@ async function check(ctx: CheckContext): Promise { // If the main sitemap has no docs URLs, try a docs-specific sitemap if (scopedSitemapUrls.length === 0 && baseUrlPath && baseUrlPath !== '/') { - const docsSitemapUrls = await fetchDocsSitemap(ctx); + const docsSitemapUrls = await fetchDocsSitemap(ctx, sitemapWarnings); if (docsSitemapUrls.length > 0) { sitemapUrls = docsSitemapUrls; scopedSitemapUrls = scopeUrls(docsSitemapUrls, effectiveOrigin, baseUrlPath); diff --git a/src/constants.ts b/src/constants.ts index fa889fc..b8c7b29 100644 --- a/src/constants.ts +++ b/src/constants.ts @@ -67,6 +67,9 @@ export const LINK_RESOLVE_THRESHOLD = 0.9; /** Maximum number of URLs to collect from sitemaps before stopping. */ export const MAX_SITEMAP_URLS = 500; +export const MAX_SITEMAP_FETCHES = 20; +export const MAX_SITEMAP_BYTES = 50 * 1024 * 1024; + /** Default llms-txt-coverage pass threshold (percentage). */ export const DEFAULT_COVERAGE_PASS_THRESHOLD = 95; diff --git a/src/helpers/get-page-urls.ts b/src/helpers/get-page-urls.ts index 84aae6b..de291a4 100644 --- a/src/helpers/get-page-urls.ts +++ b/src/helpers/get-page-urls.ts @@ -1,5 +1,5 @@ import { extractMarkdownLinks } from '../checks/content-discoverability/llms-txt-valid.js'; -import { MAX_SITEMAP_URLS } from '../constants.js'; +import { MAX_SITEMAP_BYTES, MAX_SITEMAP_FETCHES, MAX_SITEMAP_URLS } from '../constants.js'; import { getLlmsTxtFilesForAnalysis, selectCanonicalLlmsTxt } from './llms-txt.js'; import { isNonPageUrl, isMdUrl, toHtmlUrl } from './to-md-urls.js'; import { isSameSite, canonicalHost } from './host-equivalence.js'; @@ -460,14 +460,23 @@ export function filterLocaleSitemaps( const filenameLocalePattern = /\/sitemap-([a-z]{2}(?:-[a-z]{2})?)\.xml$/i; // Pattern 2: locale code in path segment (/{locale}/sitemap.xml) const pathLocalePattern = /\/([a-z]{2}(?:-[a-z]{2})?)\/sitemap[^/]*\.xml$/i; + const infixLocalePattern = /[_-]([a-z]{2}(?:-[a-z]{2})?)(?:[_-]\d+)?\.xml$/i; const locales = new Map(); const nonLocale: string[] = []; for (const url of subSitemapUrls) { - const filenameMatch = filenameLocalePattern.exec(url); - const pathMatch = pathLocalePattern.exec(url); - const match = filenameMatch ?? pathMatch; + let pathname: string; + try { + pathname = new URL(url).pathname; + } catch { + nonLocale.push(url); + continue; + } + const filenameMatch = filenameLocalePattern.exec(pathname); + const pathMatch = pathLocalePattern.exec(pathname); + const infixMatch = infixLocalePattern.exec(pathname); + const match = filenameMatch ?? pathMatch ?? infixMatch; if (match && isLocaleSegment(match[1])) { const locale = match[1].toLowerCase(); @@ -744,10 +753,30 @@ export function deduplicateVersionedUrls( return result; } +function warnOnNarrowPathPrefix( + warnings: string[], + source: string, + prefixPath: string, + matchingUrls: number, + examinedUrls: number, +): void { + if (prefixPath && examinedUrls > 0 && matchingUrls / examinedUrls < 0.01) { + warnings.push( + `${source} path prefix "${prefixPath}" matched ${matchingUrls} of ${examinedUrls} examined same-site URLs (less than 1%); consider a broader base URL.`, + ); + } +} + +interface SitemapBudget { + fetches: number; + bytes: number; +} + async function fetchSitemap( ctx: CheckContext, sitemapUrl: string, warnings: string[], + budget: SitemapBudget, ): Promise<{ urls: string[]; sitemapIndexUrls: string[] }> { if (isGzipped(sitemapUrl)) { warnings.push(`Skipped gzipped sitemap (not supported): ${sitemapUrl}`); @@ -757,7 +786,15 @@ async function fetchSitemap( try { const response = await ctx.http.fetch(sitemapUrl); if (!response.ok) return { urls: [], sitemapIndexUrls: [] }; - const xml = await response.text(); + let xml: string; + if (response.body) { + const body = await response.body(); + budget.bytes += body.bytes; + xml = body.text; + } else { + xml = await response.text(); + budget.bytes += Buffer.byteLength(xml, 'utf8'); + } return parseSitemapUrls(xml); } catch { return { urls: [], sitemapIndexUrls: [] }; @@ -766,6 +803,7 @@ async function fetchSitemap( export interface SitemapOptions { maxUrls?: number; + sitemapUrls?: string[]; originOverride?: string; pathFilterBase?: string; /** Skip URL-level locale/version refinement. Use when the caller needs raw URLs (e.g. coverage check). */ @@ -778,13 +816,33 @@ export async function getUrlsFromSitemap( opts: SitemapOptions = {}, ): Promise { const { maxUrls = MAX_SITEMAP_URLS, originOverride, pathFilterBase, skipRefinement } = opts; - const sitemapUrls = await discoverSitemapUrls(ctx, originOverride); + const sitemapUrls = opts.sitemapUrls ?? (await discoverSitemapUrls(ctx, originOverride)); const urls: string[] = []; const matchOrigin = originOverride ?? ctx.origin; // Pre-compute the path prefix so we can filter before counting against the cap. // When pathFilterBase is provided, only URLs under that prefix consume cap slots. const prefixPath = pathFilterBase ? new URL(pathFilterBase).pathname.replace(/\/$/, '') : ''; + let examinedUrls = 0; + let matchingUrls = 0; + const budget: SitemapBudget = { fetches: 0, bytes: 0 }; + + async function fetchWithinBudget(sitemapUrl: string) { + if (!isGzipped(sitemapUrl)) { + const limit = + budget.fetches >= MAX_SITEMAP_FETCHES + ? `fetch limit (${MAX_SITEMAP_FETCHES} documents)` + : budget.bytes >= MAX_SITEMAP_BYTES + ? `body byte limit (${MAX_SITEMAP_BYTES} bytes)` + : null; + if (limit) { + warnings.push(`Sitemap ${limit} reached; discovery results are partial.`); + return null; + } + budget.fetches++; + } + return fetchSitemap(ctx, sitemapUrl, warnings, budget); + } function shouldInclude(url: string): boolean { try { @@ -793,7 +851,9 @@ export async function getUrlsFromSitemap( return false; } if (!isSameSite(url, matchOrigin)) return false; - if (prefixPath) return matchesPathPrefix(url, prefixPath); + examinedUrls++; + if (prefixPath && !matchesPathPrefix(url, prefixPath)) return false; + matchingUrls++; return true; } @@ -802,10 +862,11 @@ export async function getUrlsFromSitemap( // version spectrum rather than an arbitrary prefix of the sitemap. const collectLimit = skipRefinement ? maxUrls : maxUrls * 20; - for (const sitemapUrl of sitemapUrls) { + sitemapWalk: for (const sitemapUrl of sitemapUrls) { if (urls.length >= collectLimit) break; - const parsed = await fetchSitemap(ctx, sitemapUrl, warnings); + const parsed = await fetchWithinBudget(sitemapUrl); + if (!parsed) break; // Add direct URLs (filtered to same origin and path prefix) for (const url of parsed.urls) { @@ -824,7 +885,8 @@ export async function getUrlsFromSitemap( for (const subSitemapUrl of filteredSubSitemaps) { if (urls.length >= collectLimit) break; - const subParsed = await fetchSitemap(ctx, subSitemapUrl, warnings); + const subParsed = await fetchWithinBudget(subSitemapUrl); + if (!subParsed) break sitemapWalk; for (const url of subParsed.urls) { if (urls.length >= collectLimit) break; @@ -836,6 +898,8 @@ export async function getUrlsFromSitemap( } } + warnOnNarrowPathPrefix(warnings, 'Sitemap', prefixPath, matchingUrls, examinedUrls); + if (skipRefinement) return urls; const localeFiltered = filterLocalizedUrls( @@ -1000,6 +1064,18 @@ export async function getPageUrls(ctx: CheckContext): Promise { return deduplicateVersionedUrls(localeFiltered, version); } + function scopeLlmsTxtUrls(urls: string[]): string[] { + const scoped = filterByPathPrefix(urls, filterBase); + warnOnNarrowPathPrefix( + warnings, + 'llms.txt', + new URL(filterBase).pathname.replace(/\/$/, ''), + scoped.length, + urls.length, + ); + return refineUrls(scoped); + } + /** Filter the originalMdUrls map to a subset of URLs. */ function filterOriginalMdUrls( map: Record, @@ -1015,13 +1091,13 @@ export async function getPageUrls(ctx: CheckContext): Promise { // 1. Try llms.txt links from cached results (if llms-txt-exists ran) const cached = await getUrlsFromCachedLlmsTxtWithOriginals(ctx); - let llmsTxtUrls = refineUrls(filterByPathPrefix(cached.pageUrls, filterBase)); + let llmsTxtUrls = scopeLlmsTxtUrls(cached.pageUrls); let originalMdUrls = cached.originalMdUrls; // 2. Try fetching llms.txt directly (standalone mode, llms-txt-exists didn't run) if (llmsTxtUrls.length === 0 && !ctx.previousResults.has('llms-txt-exists')) { const fetched = await fetchLlmsTxtUrls(ctx); - llmsTxtUrls = refineUrls(filterByPathPrefix(fetched.pageUrls, filterBase)); + llmsTxtUrls = scopeLlmsTxtUrls(fetched.pageUrls); originalMdUrls = fetched.originalMdUrls; } diff --git a/test/unit/checks/llms-txt-coverage.test.ts b/test/unit/checks/llms-txt-coverage.test.ts index 8896443..0a7fcac 100644 --- a/test/unit/checks/llms-txt-coverage.test.ts +++ b/test/unit/checks/llms-txt-coverage.test.ts @@ -3,6 +3,7 @@ import { http, HttpResponse } from 'msw'; import { setupServer } from 'msw/node'; import { getCheck } from '../../../src/checks/registry.js'; import { createContext } from '../../../src/runner.js'; +import { MAX_SITEMAP_FETCHES } from '../../../src/constants.js'; import type { DiscoveredFile } from '../../../src/types.js'; import { mockSitemapNotFound } from '../../helpers/mock-sitemap-not-found.js'; import { @@ -558,6 +559,37 @@ describe('llms-txt-coverage', () => { expect(result.details?.coverageRate).toBe(100); }); + test('bounds the docs-specific sitemap fallback and reports partial results (#120)', async () => { + const host = 'cov-docs-index-budget.local'; + const origin = `http://${host}`; + const docPage = `${origin}/docs/guide`; + const ctx = makeCtx(host, [docPage], '/docs'); + const requested: string[] = []; + server.use( + http.get(`${origin}/robots.txt`, () => new HttpResponse(`Sitemap: ${origin}/sitemap.xml`)), + http.get(`${origin}/sitemap.xml`, () => new HttpResponse(makeSitemap([`${origin}/about`]))), + http.get(`${origin}/docs/*.xml`, ({ request }) => { + requested.push(request.url); + return new HttpResponse( + request.url === `${origin}/docs/sitemap.xml` + ? makeSitemapIndex( + Array.from({ length: 25 }, (_, index) => `${origin}/docs/shard-${index}.xml`), + ) + : makeSitemap([docPage]), + ); + }), + ); + + const result = await check.run(ctx); + + expect(requested).toHaveLength(MAX_SITEMAP_FETCHES); + expect(result.details?.sitemapDocPages).toBe(1); + expect(result.details?.sitemapSource).toBe('/docs/sitemap.xml'); + expect(result.details?.sitemapWarnings).toEqual([ + expect.stringMatching(/Sitemap fetch limit.*partial/), + ]); + }); + test('follows docs-specific sitemap index one level deep', async () => { const host = 'cov-docs-index.local'; const docPages = [ diff --git a/test/unit/helpers/get-page-urls.test.ts b/test/unit/helpers/get-page-urls.test.ts index 25e7e76..5b7670a 100644 --- a/test/unit/helpers/get-page-urls.test.ts +++ b/test/unit/helpers/get-page-urls.test.ts @@ -16,7 +16,11 @@ import { extractVersionFromUrl, extractLocaleFromUrl, } from '../../../src/helpers/get-page-urls.js'; -import { MAX_SITEMAP_URLS } from '../../../src/constants.js'; +import { + MAX_SITEMAP_BYTES, + MAX_SITEMAP_FETCHES, + MAX_SITEMAP_URLS, +} from '../../../src/constants.js'; import { createContext } from '../../../src/runner.js'; import { fetchPage } from '../../../src/helpers/fetch-page.js'; import type { DiscoveredFile } from '../../../src/types.js'; @@ -212,6 +216,33 @@ describe('getPathFilterBase', () => { }); describe('filterLocaleSitemaps', () => { + it.each(['_', '-'])( + 'recognizes %s-delimited locale shards and query strings (#120)', + (separator) => { + const urls = [ + `https://example.com/api${separator}en${separator}1.xml?version=2`, + `https://example.com/api${separator}fr${separator}2.xml`, + `https://example.com/api${separator}en.xml`, + 'https://example.com/api-pages.xml', + 'https://example.com/api-zz-1.xml', + 'not-a-url', + ]; + + expect(filterLocaleSitemaps(urls)).toEqual([urls[0], ...urls.slice(2)]); + }, + ); + + it('recognizes infix locales in product sitemap filenames (#120)', () => { + const urls = [ + 'https://example.com/_sitemaps/dotnet_en-us_1.xml', + 'https://example.com/_sitemaps/previous-versions_fr-fr_3.xml', + 'https://example.com/_sitemaps/azure_en-us_2.xml', + 'https://example.com/_sitemaps/products.xml', + ]; + + expect(filterLocaleSitemaps(urls, 'en-us')).toEqual([urls[0], urls[2], urls[3]]); + }); + it('filters to English sub-sitemaps when locale pattern detected in filenames', () => { const urls = [ 'https://example.com/sitemap-el.xml', @@ -2343,6 +2374,213 @@ ${urls.map((url) => `${url}`).join('\n')} ]); }); + it.each([100, 101, 637])( + 'diagnoses a narrow llms.txt prefix against %i pages (#120)', + async (count) => { + const origin = 'http://narrow-llms-prefix.local'; + const content = `# Docs\n## Links\n- [Home](${origin}/docs/en/home)\n${Array.from( + { length: count - 1 }, + (_, index) => `- [Page ${index}](${origin}/docs/en/page-${index})`, + ).join('\n')}`; + const ctx = makeCtx(`${origin}/docs/en/home`, content); + ctx.options.maxLinksToTest = 1; + + const result = await getPageUrls(ctx); + + expect(result.urls).toEqual([ctx.baseUrl]); + expect(result.sources).toEqual(['llms-txt']); + expect(result.warnings).toEqual( + count > 100 + ? [expect.stringContaining(`llms.txt path prefix "/docs/en/home" matched 1 of ${count}`)] + : [], + ); + }, + ); + + it('bounds sitemap fetches when the path prefix rejects every URL (#120)', async () => { + const origin = 'http://sitemap-budget.local'; + const requested: string[] = []; + server.use( + http.get(`${origin}/robots.txt`, () => new HttpResponse(`Sitemap: ${origin}/sitemap.xml`)), + http.get(`${origin}/*.xml`, ({ request }) => { + requested.push(request.url); + if (request.url === `${origin}/sitemap.xml`) { + return new HttpResponse( + `${Array.from( + { length: 25 }, + (_, index) => `${origin}/shard-${index}.xml`, + ).join('')}`, + ); + } + return new HttpResponse(`${origin}/other/page`); + }), + ); + + const ctx = createContext(`${origin}/en-us/docs`, { requestDelay: 0 }); + const warnings: string[] = []; + const result = await getUrlsFromSitemap(ctx, warnings, { pathFilterBase: ctx.baseUrl }); + + expect(result).toEqual([]); + expect(requested).toHaveLength(MAX_SITEMAP_FETCHES); + expect(warnings).toContainEqual(expect.stringMatching(/sitemap.*limit.*partial/i)); + expect(warnings).toContainEqual(expect.stringMatching(/\/en-us\/docs.*broader base URL/i)); + }); + + it('stops after the cumulative sitemap body budget without discarding read URLs (#120)', async () => { + const origin = 'http://sitemap-byte-budget.local'; + const requested: string[] = []; + server.use( + http.get(`${origin}/robots.txt`, () => new HttpResponse(`Sitemap: ${origin}/sitemap.xml`)), + http.get(`${origin}/*.xml`, ({ request }) => { + requested.push(request.url); + if (request.url === `${origin}/sitemap.xml`) { + return new HttpResponse( + `${[1, 2, 3] + .map((index) => `${origin}/shard-${index}.xml`) + .join('')}`, + ); + } + return new HttpResponse( + `${origin}/page-${requested.length}`, + ); + }), + ); + + const ctx = createContext(origin, { requestDelay: 0 }); + const originalFetch = ctx.http.fetch; + ctx.http.fetch = async (...args) => { + const response = await originalFetch(...args); + if (!args[0].includes('/shard-')) return response; + return { + ...response, + body: async () => ({ text: await response.text(), bytes: MAX_SITEMAP_BYTES / 2 }), + }; + }; + const warnings: string[] = []; + const result = await getUrlsFromSitemap(ctx, warnings); + + expect(result).toEqual([`${origin}/page-2`, `${origin}/page-3`]); + expect(requested).toEqual([ + `${origin}/sitemap.xml`, + `${origin}/shard-1.xml`, + `${origin}/shard-2.xml`, + ]); + expect(warnings).toEqual([expect.stringMatching(/Sitemap body byte limit.*partial/)]); + }); + + it.each(['empty', 'http-error', 'network-error'])( + 'counts %s sitemap attempts across all roots in raw coverage mode (#120)', + async (responseKind) => { + const origin = 'http://sitemap-failed-budget.local'; + const roots = Array.from({ length: 25 }, (_, index) => `${origin}/map-${index}.xml`); + const requested: string[] = []; + server.use( + http.get(`${origin}/*`, ({ request }) => { + requested.push(request.url); + if (responseKind === 'network-error') return HttpResponse.error(); + return new HttpResponse('', { + status: responseKind === 'http-error' ? 404 : 200, + }); + }), + ); + const ctx = createContext(origin, { requestDelay: 0 }); + const warnings: string[] = []; + + const result = await getUrlsFromSitemap(ctx, warnings, { + sitemapUrls: roots, + skipRefinement: true, + }); + + expect(result).toEqual([]); + expect(requested).toEqual(roots.slice(0, MAX_SITEMAP_FETCHES)); + expect(warnings).toEqual([expect.stringMatching(/Sitemap fetch limit.*partial/)]); + }, + ); + + it('shares the fetch budget across multiple sitemap indexes (#120)', async () => { + const origin = 'http://sitemap-shared-budget.local'; + const roots = [`${origin}/first.xml`, `${origin}/second.xml`]; + const requested: string[] = []; + server.use( + http.get(`${origin}/*`, ({ request }) => { + requested.push(request.url); + if (roots.includes(request.url)) { + return new HttpResponse( + `${Array.from( + { length: 10 }, + (_, index) => `${request.url}/shard-${index}.xml`, + ).join('')}`, + ); + } + return new HttpResponse(''); + }), + ); + const ctx = createContext(origin, { requestDelay: 0 }); + const warnings: string[] = []; + + expect(await getUrlsFromSitemap(ctx, warnings, { sitemapUrls: roots })).toEqual([]); + expect(requested).toHaveLength(MAX_SITEMAP_FETCHES); + expect(requested[0]).toBe(roots[0]); + expect(requested[11]).toBe(roots[1]); + expect(warnings).toHaveLength(1); + }); + + it.each([1, MAX_SITEMAP_FETCHES])( + 'does not warn when the URL cap or end of input stops at %i documents (#120)', + async (maxUrls) => { + const origin = 'http://sitemap-complete-budget.local'; + const roots = Array.from( + { length: MAX_SITEMAP_FETCHES }, + (_, index) => `${origin}/map-${index}.xml`, + ); + const requested: string[] = []; + server.use( + http.get(`${origin}/*`, ({ request }) => { + requested.push(request.url); + return new HttpResponse(`${origin}/page`); + }), + ); + const ctx = createContext(origin, { requestDelay: 0 }); + const warnings: string[] = []; + + const result = await getUrlsFromSitemap(ctx, warnings, { + sitemapUrls: roots, + maxUrls, + skipRefinement: true, + }); + + expect(result).toHaveLength(maxUrls); + expect(requested).toEqual(roots.slice(0, maxUrls)); + expect(warnings).toEqual([]); + }, + ); + + it('uses same-site URLs only for the narrow-prefix ratio (#120)', async () => { + const origin = 'http://sitemap-prefix-ratio.local'; + server.use( + http.get( + `${origin}/sitemap.xml`, + () => + new HttpResponse( + `${origin}/docs/page${Array.from( + { length: 200 }, + (_, index) => `https://other.example.com/other/${index}`, + ).join('')}invalid-url`, + ), + ), + ); + const ctx = createContext(`${origin}/docs`, { requestDelay: 0 }); + const warnings: string[] = []; + + const result = await getUrlsFromSitemap(ctx, warnings, { + sitemapUrls: [`${origin}/sitemap.xml`], + pathFilterBase: ctx.baseUrl, + }); + + expect(result).toEqual([`${origin}/docs/page`]); + expect(warnings).toEqual([]); + }); + it('respects maxUrls cap when following sitemap index sub-sitemaps', async () => { mockSitemapNotFound(server, 'http://cap-index.local'); server.use( diff --git a/working-notes/page-discovery-notes.md b/working-notes/page-discovery-notes.md index 6efd8ae..05e938c 100644 --- a/working-notes/page-discovery-notes.md +++ b/working-notes/page-discovery-notes.md @@ -247,6 +247,134 @@ Nested-index regressions assert that discovery fetches only the index, not its page candidates. Markdown and negotiation regressions also pin the requests made for the formerly skipped pages. +### September 2026: bounded sitemap walks and infix locales (issue #120) + +A narrow Microsoft Learn base URL (`/en-us/docs`) matched no sitemap +pages, so the accepted-URL cap never fired. The index contained thousands +of large shards. The locale filter also missed product filenames such as +`dotnet_en-us_1.xml` and `previous-versions_fr-fr_3.xml`. + +Decisions: + +- Recognize a filename locale token delimited by `_` or `-`, optionally + followed by a numeric shard. Match the parsed pathname, ignoring query + strings. Validate with the existing locale-code helper, require two + distinct locales before filtering, preserve non-locale sitemaps, and + retain the existing preferred-locale and fallback rules. +- Each sitemap walk has a shared ceiling of 20 sitemap fetch attempts, + including roots, indexes, children, failed responses, and empty responses. + This is independent of accepted URLs. Gzipped sitemaps remain unsupported + and do not consume fetch slots. Robots discovery is separate; HTTP + redirects and retries retain their existing behavior, so 20 fetch calls + is not a promise of 20 wire requests. +- Stop starting additional sitemap fetches after successfully read bodies + total 50 MiB of decoded bytes. This leaves room for one of the reported + roughly 46 MB shards plus its index, while preventing hundreds of such + downloads. Use `HttpResponse.body()` for served-byte accounting, or UTF-8 + text length for custom clients without that method. +- The byte budget is checked between responses, not during streaming. + The final body can overshoot it, and a failed partial read has no byte + count. It is not a per-response memory ceiling or a wall-clock guarantee; + the existing HTTP timeouts still apply. A hard streaming cap would require + a separate HTTP API change, including canonical-origin rewriting. +- When a budget prevents another fetch, retain already-collected URLs and + warn that discovery is partial. Do not issue a budget warning merely + because the last input or the existing accepted-URL cap ended the walk. + Normal locale/version refinement and sampling limits remain unchanged. +- Share the bounded walker with coverage's docs-specific sitemap fallback + through an optional explicit sitemap-root list. Coverage and page discovery + can run separate walks, each with its own budget; this is not a scan-wide + quota. Partial coverage results retain `sitemapWarnings` and should not be + treated as exhaustive coverage of the site. +- Warn when a non-root path prefix matches fewer than 1% of examined + same-site URLs, before locale/version refinement. Include the prefix, + counts, and a suggestion to use a broader base URL. Apply this to both + sitemap and llms.txt discovery. Empty sources and root prefixes do not + warn; zero matches from a nonempty source does. Do not broaden the scope + automatically. + +Rejected alternatives: relying on locale filtering alone (one locale can +still have hundreds of shards), stopping after a few nonmatching shards +(later shards may contain the requested product), and counting only +successful or accepted pages (the original unbounded-walk failure). +No per-page probes, deeper traversal, or configurable budget API were added. + +Regression tests pin locale forms, fetch and byte limits, shared budgets +across roots and indexes, empty/error responses, raw coverage mode, fallback +warning propagation, normal termination, and the strict 1% boundary. +The initial locale and unbounded-walk tests both failed before the fix. + +## Documentation at scale: design considerations + +Drafting `docs/documentation-at-scale.md` after the #120 live reproduction +exposed several distinctions that the current CLI makes easy to miss. These +are design questions, not new configuration options or requirements to +expand the bounded-walk fix. + +### Prioritize completeness and source selection + +1. **Make discovery completeness machine-readable.** Budget exhaustion is + currently communicated through warning strings. A completed scan, an + incomplete discovery set, and a base-page fallback are different states. + Consider structured metadata for sources, stop reasons, examined and + retained counts, and the scope of each walk. Preserve it in reports and + show it prominently enough that a high page-quality score cannot be + mistaken for a complete product assessment. Avoid claiming an exhaustive + set even when a budget was not reached: collection caps, omitted nested + indexes, and locale/version refinement still affect the corpus. +2. **Define a product-specific sitemap selection contract.** The CLI/config + can select a published llms.txt with `llmsTxtUrl` but has no equivalent + sitemap selector. A correct product prefix can still find nothing before + the global index budget runs out. The helper's explicit sitemap roots + added for coverage fallback do not settle a user-facing API. Any selector + needs consistent behavior across discovery and coverage, origin rewriting, + scope and locale rules, and explainable fallback when the selected source + fails. An explicit narrow source is preferable to guessing products from + filenames or silently increasing traversal depth. + +### Keep selection, requests, and scoring distinct + +- **Separate page selection from index work in the interface.** Curated + sampling is repeatable page selection, not a request allowlist. Coverage + and index link checks have their own work. Documentation now corrects the + old "skips discovery entirely" wording. A future resolved-config summary + or check-scope summary could show this before network work begins, without + promising an exact request count. Prefer visibility over adding named + profiles until teams demonstrate stable profile needs. +- **Choose an explicit policy for incomplete CI evidence.** Warning-level + conditions do not normally fail the CLI. A team may want incomplete + discovery to block a discovery audit while allowing its curated page + regression job to run. This should be an explicit policy built on + structured completeness, not matching warning text or treating missing + evidence as a page-quality failure. Consider exit behavior, skipped checks, + JSON consumers, and scoring together before choosing a flag or default. +- **Distinguish a scope from a network boundary.** Today the base path and + locale/version preferences filter discovered candidates, while curated + entries are used directly. Redirects, Markdown candidates, and index link + checks can reach other URLs. An enforced allowlist would be a separate + feature with explicit behavior for those cases; do not imply that a base + URL already provides it. Keep this separate from page/non-page URL + classification and Markdown URL mapping. +- **Consider shared budgets and reuse before higher limits.** Discovery and + coverage can repeat sitemap work in separate walks, and parallel product + jobs do not share a rate limiter. A scan-wide budget or cache needs to + respect origin rewriting, path scope, raw coverage versus refined samples, + and cancellation. A hard response-byte cap also needs HTTP-layer support; + the current between-response budget is not that guarantee. Larger default + limits would not establish representative sampling across products. +- **Report enough context for comparison.** Config-selected checks, curated + pages, tags, environment, and AFDocs/scoring versions affect what a score + means. Define how those inputs travel with a report before adding + cross-product score aggregation. Tags currently group selected results; + they do not implement stratified discovery or statistically representative + sampling. + +The published guide uses existing controls only: product-owned configs, +explicit page-check selection, separate discovery audits, and review of +warnings and tested pages. Future implementations should keep the existing +no-per-discovered-page-probe and bounded-depth invariants, and test request +accounting and partial-result semantics alongside any new option. + ## Invariants - Discovery emits page URLs; `.md` candidates are derived per check.