Conversation
The shipped workflow imported tests at @v1, and this repo has no tags, so `gh aw add` failed at compile. Nobody could follow the README to a working state. Fixing that pulled in the surrounding UX. Install is now one command. `gh aw add canonical/user-docs-testing/docs-testing` already worked against the existing repo layout, it just wasn't documented; it also records a source ref, so `gh aw update` handles upgrades. Imports are fixed and `tests:` in the config decides what runs, so installing no longer means editing the workflow. A composite action replaces the commented-out CI block, the tool-repo checkout (spelled two different ways), and the hand-written script invocations. Results gain an explicit error state. The old orchestrator turned a crashed command, a missing results file, or malformed JSON into a warning and zero findings, which is how a broken tool reported a clean pass. Outcomes are now pass / warn / incomplete / fail / error, with error outranking everything. Also fixed: a missing required source reported "warn" when no source_map existed, a docs glob matching zero files could pass, and source evidence inherited the enclosing repo's HEAD via `git -C`, so it attested commits for directories that were never cloned. Config is validated once, up front, with a location and a suggested fix. Built-in checks are declared (`uses: undocumented-surface`) instead of invoked as script paths, and source evidence is now automatic rather than a test users configure. Breaking config change; `validate` names each field to rename. User commands no longer run through a shell. Config format changed before any release, so no compatibility shim.
Two questions kept coming up that the docs could not answer: how do I use something other than Copilot, and what is this lock file I keep having to commit. Both had good answers already — they just were not written down. Add docs/reference/how-it-works.md, covering why Actions needs a compiled lock file, what gets pinned when, and what actually happens during a run. The important part is the table showing that editing docs-testing.config.yml needs no compile at all, since the config is read at run time; day-to-day work never touches generated files. Add docs/reference/scheduling.md. gh-aw accepts friendly schedules like `weekly on monday` and scatters them per repository so many repos do not all fire at once, but monthly needs raw cron. Running a cheap check often and an expensive review occasionally means installing the workflow twice with separate configs, which the action's `config` input already supports. Engine selection moves to install time: `gh aw add ... --engine claude` writes and compiles the workflow in one command, so nobody has to hand-edit frontmatter and recompile just to avoid Copilot. Writing the docs surfaced three defects. A link to the gh-aw engine reference pointed at githubnext.github.io, which serves the homepage but 404s every deep path; the live host is github.github.io, and all ten upstream links are now verified. An exit-status check's output was captured into the results file but never rendered, so a linter that printed exactly which file was broken showed up as "(no file)" — the zero-effort adapter looked useless. And migrating an older config gave `type` the fix "rename it to `uses/run (the kind of test is now inferred)`", which is not a sentence; removed keys now explain themselves, and a test named after a built-in is told to add `uses: <that name>`. Fold selftests/test_scope.py and test_source_evidence.py into test_contracts.py. They imported ContractTest across files, which only worked because unittest discovery happened to put that directory on the path. Drop a CI step that re-asserted in YAML what the suite already covers.
The reference tests had already been validated behaviorally, and I had rewritten them: splitting the shared rules into a reference-common.md and cutting each test down to the question unique to it. That is a reasonable structure, but it re-rolls the dice on prompt behavior that was known to work, and nothing in this branch was going to re-verify it. Not a trade worth making for a change whose purpose is user experience. reference-review.md and reference-completeness.md are back to their original content. Each differs by exactly one line: a link to RESULTS-SCHEMA.md, which no longer exists and now points at docs/reference/results.md. reference-common.md is deleted, and the conclusion rules it held move back inline into the workflow prompt where they were before. tests/deterministic/ is restored byte-identical. It also turns out to be load-bearing: reference-completeness.md links to undocumented_surface.py by relative path, so removing it would have forced a second edit to a file that should not be edited. The config key goes back to `targets:`. Renaming it to `docs:` was my churn, and it left the restored prompts describing a key the config no longer had. Reverting costs nothing a user sees and keeps the prompts correct as written; the resolved plan in results/all.json now emits `targets` too, so what the agent is handed matches the vocabulary it was tested against. None of this changes how the tool reaches a consumer. A repository still needs a workflow and a config and nothing else: the reviews arrive through `imports:`, and the deterministic checks through the composite action, so no script is ever copied into a consuming repository.
Remove the `uses: undocumented-surface` built-in and its duplicate copy of the matching logic. Deterministic checks are your own command again, and the scripts under tests/deterministic/ are documented as demonstrations.
…c that generated content can be skipped
Collaborator
Author
|
Dev part should be done now - need to cleanup (docs, comments, examples, etc) |
…h-aw The README led with the two agentic reviews, which read as an AI product rather than a workflow that runs documentation checks. It now says what it is first: a wrapper around gh-aw that runs checks from one config file, with deterministic checks of your own alongside the two that ship. Engine and schedule pages reproduced gh-aw's tables and would have drifted out of date unnoticed. They now link upstream at the point of use and keep only what is ours. The Copilot PAT fallback is gone: it does not apply here, and it was the least supported claim on the page. Versioning was a page for two facts. The pinning is in the README status note, and the interface-stability rules moved next to the results schema they describe, where someone writing a check will actually read them. Adds an architecture map and a short contributing guide, both missing, and a tutorial that walks the minimal example through warning, pass, and incomplete. Flattens tone throughout: no stakes-setting, no reassurance. Fixes paths left stale by the Diataxis move, four of them in the workflow that gets copied into user repositories, where a repo-relative docs path resolves to nothing; those are now URLs. Corrects the Landscape example from six repositories to four, one private. Removes the frontmatter emoji, confirmed optional by compiling without it.
The example configs carried their explanation in comments, which made them cluttered to read and tedious to adopt. They are now plain YAML with a two-line header; the three things only the comments knew moved into the configuration reference. Drops the top-level annotated config: a third Landscape configuration overlapping the worked example, and the one that had drifted, still giving `auth:` to two public repositories. Its removal left five fields with no YAML form, so the reference gained snippets for them. The self-test workflow validated that file by name and would have failed. Renames examples/landscape to examples/full-product, since the pairing that matters is minimal against a whole product. Its documentation is not in this repository, so the README now says so and carries the ownership table that source_map encodes. Removes fetch-fixtures.sh: nothing uses it, and CI has always validated this example without the fetched content.
Repoints what still referenced examples/landscape, and drops the deleted top-level config from the self-test workflow, which named it directly. The configuration reference gains what those comments knew — the generated.mode values, that an area may have several owners, and that a test's `sources` limits what it consults without deciding ownership — plus snippets for the five fields no example shows any more. The full-product README no longer promises a fetch script, and says plainly that the documentation it refers to lives elsewhere.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Including a description of what's in here - reviewers aren't expected to review/confirm every file, but please read this description first for an overview of what's going on.
NOTE TO REVIEWERS: I haven't finished polishing this PR and its documentation, so keep that in mind as you review it. I'm expecting to make more docs changes. Feel free to add any suggestions if you want, but note that the docs especially haven't had a thorough pass (except for the README) - there isn't wrong information, but the quality isn't polished. But I'm done with the dev work that's expected in this PR, so I won't make any more changes there unless something comes up in review.
And I'm totally fine leaving this as a PR for now until others are ready for it to be merged - you can run against this specific branch in GitHub Actions - there's no urgency to get it merged as it can be tested fine for now.
I've left the tutorial in docs/tutorial so it doesn't mess anything up for others, but later that would have to get removed since it's an example, and the docs/ directory has real (in progress 😊) user documentation now.
Overview
I've restructured this project into an installable Python package (
docs_testing) plus a GitHub Actions workflow, replacing the previous script collection. It wraps gh-aw: the workflow reads a config file in the target repo, runs whatever checks it declares, and reports one Check Run.I'll give a super high-level of it here, but to understand what's going on with the project itself, I suggest looking at the README and the architecture file ("architecture" as in it describes what all is in the repository).
BASICALLY this project is a wrapper around gh-aw, run as a GitHub Actions workflow. A user would add the upstream project with
gh aw add canonical/user-docs-testing/workflows/docs-testing.mdand manually create their own config file (YAML) that specifies their docs, code source repos/locations, other details, etc. In addition to reporting failures/warnings, this project also constrains what the agents are allowed to report (e.g., it notes if any code sources couldn't be reached, if certain docs couldn't be verified like if there was no source to compare to, etc..), so it can also report docs as unverified, rather than assuming a "pass" when there's no source to confirm.There are currently two shipped agentic tests:
reference-review: flags documentation that contradicts the product.
reference-completeness: flags product surface the documentation never mentions.
Deterministic checks (no AI) can be added too on a project-specific basis. Those run first, so the agentic tests won't cover the same thing. This repo includes one deterministic check as an example (so I could test to make sure it would run). Users who want deterministic checks would have to supply their own though.
Everything in this PR has been tested in CI (in my example tester project). I tested both agentic reviews plus the sample deterministic check included in this source repo. The tests ran against the Landscape docs and its code repos, which included both public and private repos. (I can demo the report to you in our next meeting)
And there are two example configs:
minimal: Super clean and basic, users could copy or play around with this one, it shows a lightweight config.
full-product: A real example that's customized for a project (Landscape). This one is meant to be read/referenced, not copied into a user's own docs repo.
TESTING
When I tested, I used this file, note the branch as well as it's not main, and in the Actions tab, I ran the test manually, specifying to run it against that specific branch.
If you have a setup that uses what's in
mainin this repo, I'm not totally sure what needs to change because I haven't tested that for awhile, but the parts that need to change in the testing config file are the imports and uses lines, which need to specify thedocs-testing-product-uxbranch from this PR. This PR also introduces changes to how the engine is specified. (you can also runpipx install git+https://github.com/canonical/user-docs-testing@docs-testing-product-uxthendocs-testing validateto see what to change - I haven't tried this though aside from having Copilot test it, but it's a read-only command anyway so it can't ruin anything)The changes introduced in this PR shouldn't impact your tutorial testing at all, and actually by design your connectivity shouldn't get messed up if we merge this PR due to the lockfile stuff, so you only update once you re-compile or update. However, I'm fine leaving this PR open for awhile though as you wrap up that work (if you don't want to focus on switching up the infra stuff yet), just in case you accidentally re-compile / update (or something in the project is off), I don't want to mess up your connectivity while you finish up the tutorial agentic test.
Additional changes (below is Ai-generated, just listed it for myself, other things I did in this PR)
Fixed a bug where composite action steps run under bash -e, so a non-zero exit from the runner killed the agentic review before it could report — on exactly the runs that most needed reporting. The exit code is captured now instead of being fatal.
Added severity tiers (error/warning) and a conclusion ladder that picks the Check Run's success/neutral/failure/action_required. This is what actually decides pass/fail, not just whether a report got posted.
Added source_evidence: records what was actually checked out (directory, commit, file count), so a review can't claim to have checked a source that was never cloned.
42 self-tests, run in CI on every change.
Known limitations, since this is beta: findings can vary between runs (no finding doesn't mean nothing's wrong), and around 22 files in one review, the agent has started delegating to sub-agents that under-report. Both documented, neither fixed.
Testing docs and code in the same repo needs a second checkout under sources/. Works, but undocumented and easy to miss.