Clear 568 structural findings across 32 skills - #228
Open
bpmadde wants to merge 44 commits into
Open
Conversation
Adds the remediation contract the audit gate is built against: - specs/01-skill-structure.md, specs/02-audit-findings.md, specs/03-genie-code-skill.md - the structural contract and the audit's per-finding calibration targets. - IMPLEMENTATION_PLAN.md - measured per-finding counts, landing order, and the open policy questions. - PROMPT_plan.md, PROMPT_build.md, loop.sh - the loop harness itself. CLAUDE.md carries an uncommitted edit that predates this branch (the plugin-manifest section condensed into 'Generated files - never hand-edit'); it is included here rather than left dirty for every future run. No skill content and no generated artifact is touched. Signed-off-by: Brian Madden <bpmadde@icloud.com>
`scripts/skills.py validate` checks the plugin plumbing only — none of its
19 checks looks at skill content. This adds the content-side counterpart, so
"is this finding fixed?" is a count rather than a judgement call, and every
remaining finding class gets a one-command backpressure gate before its
sweep starts.
python3 scripts/audit_check.py # rollup, one count per ID
python3 scripts/audit_check.py --only PD-5 # exit 0 when that ID is 0
python3 scripts/audit_check.py --only PD-5 --details
22 findings are registered. Every must-fix count reproduces the remediation
plan's measured numbers exactly: PD-6 125, PD-5 228, PD-5b 12,
SPEC-10a-cross 90, SPEC-10a-intra 21, SPEC-10a-prose 3, SPEC-10b 64, NEW-A 6,
NEW-C 4, PD-4a 3, PD-4c 3, PD-1 3, PD-2 3, DESC-1 6; blocked PD-4b 19,
TOK-5 21, COMPAT-1 7; advisory in-fence 81, NEW-B 1, PD-8 30, MNT-6 1.
Resident set 2,975 tokens stable / 3,199 all-in, also exact.
No skill content changes. Stdlib-only and no `raise`/`assert`, matching
scripts/skillsgen/ house style (`check_*(repo_root) -> list[str]`, the caller
prints); reuses `iter_all_skill_dirs`. Not wired into `validate` — that is
MNT-1b, for the last sweep PR, because the house test pattern
`assertEqual(check_x(_REPO), [])` would fail while any class is above zero.
Three counting definitions had to be settled for the numbers to agree, each
documented in the module docstring:
- PD-6 covers root-level reference files, not just `references/`. The four
such files are all over 100 lines with no TOC; scoping the check to
`references/` is the exact blind spot the root-file finding describes.
- SPEC-10a-cross counts links, not `../` occurrences — a `../../` target
matches the traversal regex twice. 90 links is 111 occurrences.
- SPEC-10b excludes markdown link *labels* (display text that routes
correctly) and basenames resolving at the skill root (those belong to the
root-file finding). Without both, the class reads 167 instead of 64.
Skills touched: none. Reconciliations for MNT-6 (spec says 2 skills bundle
scripts without stated intent; databricks-unstructured-pdf-generation does
state it at SKILL.md:17 and :43, so the count is 1), TOK-5 (no basis
reproduces 30/10) and five others are recorded in
specs/02-audit-findings.md; IMPLEMENTATION_PLAN.md carries the live baseline
and AGENTS.md the commands.
Gates: audit_check runs clean, `skills.py validate` exits 0, the unit suite
is 143 passing (118 pre-existing + 25 new), no generated-artifact drift.
Signed-off-by: Brian Madden <bpmadde@icloud.com>
Re-measured rather than transcribed, which corrected two figures in the reconciliation appended with the audit gate: - per-skill rounding of the resident set gives 2,974 / 3,199 on the checker's basis (folded block scalars, unescaped "). The 2,980 / 3,205 variant comes from leaving databricks-dbsql's >- description unfolded; both are now stated explicitly, since only sum-then-round-once reproduces the audit's 2,975 / 3,199. - the 13 traversals outside link targets are 11 plain prose + 2 inside link labels, of which 3 are defects. Spec text only; no checker or skill content changes. Signed-off-by: Brian Madden <bpmadde@icloud.com>
Signed-off-by: Brian Madden <bpmadde@icloud.com>
Signed-off-by: Brian Madden <bpmadde@icloud.com>
Signed-off-by: Brian Madden <bpmadde@icloud.com>
Signed-off-by: Brian Madden <bpmadde@icloud.com>
Signed-off-by: Brian Madden <bpmadde@icloud.com>
Signed-off-by: Brian Madden <bpmadde@icloud.com>
The audit's headline counts and the checker's counts disagree in *units*:
the audit counts skills where the checker counts links, occurrences where
it counts files. A plan item reading "PD-5: 17 skills -> 0" cannot be gated
against a row reporting 228 links, and three must-fix rows (PD-5b, NEW-A,
NEW-C) had no spec entry at all.
Every Count: line now carries the checker's number, its unit named
explicitly, and the audit_check.py row ID that reports it -- the same ID
--only accepts. Where the audit's figure differs it is preserved in
Reconciliation rather than discarded.
Structural changes:
- PD-4 split into PD-4a (3 occurrences) / PD-4b (19 skills, blocked) /
PD-4c (3 files); one headline could not serve three units and two
severities.
- PD-1/PD-2/PD-3 split 3 / 3 / 4; the ceilings bind independently, so
the audit's 4 is the PD-3 rollup, not either gate.
- New entries for PD-5b (12 files), NEW-A (6 links), NEW-C (4 files),
NEW-B (1 link, advisory), SPEC-10a-self-parent (3 occ, exempt).
- SPEC-10a-prose restated 3 -> 0: 2b9807a exempted the self-parent
../SKILL.md occurrences under D8, so the spec had gone stale.
- Per-skill lists corrected against the gate (PD-5 pipelines 91 not 103;
TOK-5 lakeflow-connect 1 not 9).
New reconciliations for PD-5, PD-5b, DESC-1, COMPAT-1, PD-4, PD-1/2/3 and
NEW-A/B/C, in the style SPEC-10b and the in-fence exemption established.
COMPAT-1's 7 decomposes as 5 excess shapes + 2 body-vs-frontmatter
conflicts; recorded with it is a checker question -- (absent) sorts first
and is charged as the target shape -- deliberately not fixed here, since
this pass restates counts and must not move one.
Spec-alignment only, not remediation. No file under skills/ or
experimental/ is touched and no count changed: audit_check.py output is
byte-identical before and after (568 must-fix, resident set 2,975/3,199).
Baseline measured against upstream/main (50ccd08), which the corpus
matches exactly -- skills/ tree 4675ac0, experimental/ tree 32b7073.
IMPLEMENTATION_PLAN.md carries the same stale counts; left alone to keep
this commit single-concern and cherry-pickable.
Verified: audit_check.py diff empty; skills.py validate exit 0; 143 tests
pass; every must-fix and blocked row now named with its checker count.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4
Signed-off-by: Brian Madden <bpmadde@icloud.com>
91 links across 23 reference files pointed at sibling reference files. The second hop is invisible to a partial read (head -100), so the target was effectively unrouted even though a link existed. Each link is replaced by the target's plain name -- basename minus .md, keeping any #anchor so the pointer still names the section it meant. Fenced code is untouched: the checker exempts it under D1 and the fenced paths are working examples. Routing moves to SKILL.md, which is where a reference is supposed to be reachable from. Its Reference Index already linked 31 of 33 files; this adds the two it missed -- python-basics.md and sql-basics.md -- as a "Language primers" group, each with a stated condition for when to read it. Those two were reachable only via a second hop from 2-rapid-iteration-with-cli.md, which is exactly the defect PD-5 describes, and they were also the skill's two PD-4c orphans. One readability fix beyond the mechanical strip: streaming-patterns.md:94 read "use real-time-mode" after the substitution, turning a product name into a filename; restored to "use Real-Time Mode -- see real-time-mode". Backpressure: PD-5 228 -> 137 repo-wide, databricks-pipelines 91 -> 0 PD-4c 3 -> 1 repo-wide, databricks-pipelines 2 -> 0 databricks-pipelines must-fix 119 -> 26 No other finding count moved in either direction. skills.py validate and test_repo_bundle_is_canonical fail at this commit: plugins/** is a generated copy of skills/ and is now stale. Regenerated in a separate self-heal commit so this content diff stays reviewable -- the same split upstream uses (50ccd08, 76d4fdc). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4 Signed-off-by: Brian Madden <bpmadde@icloud.com>
plugins/** is a generated copy of skills/; the PD-5 sweep in the preceding commit left it stale. Mechanical regeneration only -- `python3 scripts/skills.py generate`, no hand edits. 96 files, 24 per provider (claude, codex, copilot, cursor), mirroring the 24 files changed under skills/databricks-pipelines/. Nothing else moved: manifest.json, the four marketplace catalogs, the hook wiring, and the routing files all regenerate byte-identical. Split out from the content commit so the 24-file PD-5 diff stays reviewable rather than being buried in 120 files -- the same split upstream uses (50ccd08, 76d4fdc). After this commit: skills.py validate exits 0 and all 143 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4 Signed-off-by: Brian Madden <bpmadde@icloud.com>
Counts for the two findings this branch moved: PD-5 228 -> 137 (14 skills left), PD-4c 3 -> 1. Branch table gains the completed pipelines row and rescopes the remaining PD-5 branch to 137. Adds a "Sweep recipe" validated on 91 links, so the next 14 skills do not rediscover it: keep the #anchor when stripping, skip fenced code, route the un-linked files from SKILL.md with a read condition, and read the diff -- the mechanical pass demoted "Real-Time Mode" to a filename in one spot. Two findings that overlap in practice: a reference reachable only by a second hop is both a PD-5 defect and a PD-4c orphan, which is why clearing pipelines' 91 links also cleared its 2 orphans. Expect that pairing elsewhere. Records the bundle conflict: any skills/** change stales plugins/**, so skills.py validate cannot pass without `scripts/skills.py generate`, while GEN-1 forbids the branch touching plugins/**. Both cannot hold in one commit; the resolution is upstream's own split -- content first, then a separate self-heal commit. Defers one thing noticed but not owned here: the count table's SPEC-10a-prose row still reads 3 where the checker reports 0 since 2b9807a. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4 Signed-off-by: Brian Madden <bpmadde@icloud.com>
…dashboards 48 links across 9 reference files pointed at sibling reference files -- 29 in databricks-unity-catalog, 19 in databricks-aibi-dashboards. The second hop is invisible to a partial read (head -100), so the target was effectively unrouted even though a link existed. Each link becomes the target's plain name -- basename minus .md, keeping any #anchor so the pointer still names the section it meant. Fenced code is untouched: the checker exempts it under D1 and the fenced paths are working examples. No routing changes needed. Both SKILL.md files already link every reference directly with a stated condition -- unity-catalog's seven-row reference table (SKILL.md:47-53) and aibi-dashboards' "Reference files" table (:332-336) -- so nothing was left reachable only by the second hop. PD-4c stays at 1. Two readability fixes beyond the mechanical strip, both the product-name demotion the pipelines sweep hit at streaming-patterns.md:94: 3-securables-ddl.md:57 read "a path under a registered 2-external-locations", turning the UC concept "external location" into a filename; restored to "a path under a registered external location (see 2-external-locations)". 1-widget-specifications.md:401 carried two links whose labels were the widget names Heatmap and Choropleth Map; the strip left two bare filenames in a row. Restored the names with the pointer parenthesised. Backpressure: PD-5 137 -> 89 repo-wide, unity-catalog 29 -> 0, aibi-dashboards 19 -> 0 databricks-unity-catalog must-fix 47 -> 18 databricks-aibi-dashboards must-fix 33 -> 14 No other finding count moved in either direction. skills.py validate and test_repo_bundle_is_canonical fail at this commit: plugins/** is a generated copy of skills/ and is now stale. Regenerated in a separate self-heal commit so this content diff stays reviewable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4 Signed-off-by: Brian Madden <bpmadde@icloud.com>
`scripts/skills.py generate` after the PD-5 sweep of unity-catalog and aibi-dashboards. 36 files = the 9 changed reference files x 4 provider bundles. No hand edits; `skills.py validate` is green again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4 Signed-off-by: Brian Madden <bpmadde@icloud.com>
…apps
PD-5b: `skills/databricks-apps/references/appkit/` was the repo's only two-level
reference directory. Its 12 files move to `references/appkit-*.md` (git mv, no
content rewrite beyond the link repointing below). `appkit-sdk.md` keeps its
name -- it already carried the prefix, and `appkit-appkit-sdk.md` reads as a
typo.
Flattening changes what relative paths mean, so three sets of links move with
the files. All were verified to resolve after the move:
- 20 links in `databricks-apps/SKILL.md`, `references/appkit/X.md` ->
`references/appkit-X.md`.
- 4 inbound cross-skill paths that would otherwise dangle and raise NEW-A:
`databricks-model-serving/SKILL.md:220` and
`databricks-lakebase/references/{pgvector,off-platform,connectivity}.md`.
They keep their `../` form -- that is SPEC-10a-cross's defect to fix, not
this branch's, and retargeting them here would hide it.
- 2 links inside `appkit-lakebase.md` that pointed out of the skill with
`../../../`; one level shallower after the move, so `../../`.
PD-5: the skill's 19 reference-to-reference links become the target's plain
name -- basename minus .md, anchor kept. Fenced code untouched (D1).
All 15 reference files remain linked directly from SKILL.md with a stated
condition; the "Phase / READ BEFORE proceeding" table carries 12 of them.
One link needs naming. `appkit-proto-first.md:306` pointed at
`references/plugin-contracts.md`, which exists nowhere in the repo -- the
advisory NEW-B row, held out of NEW-A because which file it meant is a
maintainer decision. PD-5 requires the link gone either way, so it takes the
same mechanical strip as the rest and reads `plugin-contracts` in prose. That
names what the author meant without asserting a path, and the decision stays
open. NEW-B therefore reads 0: the defect is now prose, not a broken link.
Backpressure:
PD-5b 12 -> 0 repo-wide; the nested directory is gone
PD-5 89 -> 70 repo-wide, databricks-apps 19 -> 0
SPEC-10a-intra 21 -> 19; the two `../platform-guide.md` links from inside
appkit/ became sibling links when the directory flattened
NEW-B (advisory) 1 -> 0 as described above
databricks-apps must-fix 49 -> 16
SPEC-10a-cross holds at 90 and NEW-A at 6 -- the rename broke no path. No
finding count increased.
skills.py validate and test_repo_bundle_is_canonical fail at this commit:
plugins/** is a generated copy of skills/ and is now stale. Regenerated in a
separate self-heal commit so this content diff stays reviewable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4
Signed-off-by: Brian Madden <bpmadde@icloud.com>
`scripts/skills.py generate` after the appkit flatten and the databricks-apps PD-5 sweep: 12 renames plus the changed reference and SKILL.md files, x 4 provider bundles. The generator prunes the stale `references/appkit/` directories, so no bundle copy of the nested layout survives. `manifest.json` moves with them -- it carries the per-skill file list, which the renames change. No hand edits; `skills.py validate` is green again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4 Signed-off-by: Brian Madden <bpmadde@icloud.com>
45 links across 11 reference files pointed at sibling reference files --
databricks-spark-structured-streaming 15, databricks-metric-views 13,
databricks-lakebase 9, databricks-ml-training 8. The second hop is invisible to
a partial read (head -100), so the target was effectively unrouted even though
a link existed.
Each link becomes the target's plain name -- basename minus .md, keeping any
#anchor. Fenced code untouched (D1). The three exempt `../SKILL.md` links in
metric-view-advisor.md are self-parent references, not reference-to-reference,
and are left alone.
Routing: `databricks-lakebase/references/medallion-from-cdc.md` was reachable
only through a second hop from lakehouse-sync.md and synced-tables.md, so
stripping those links would have stranded it. It is now the sixth entry in the
skill's Reference docs list (SKILL.md:50) with a stated read condition. It was
also the repo's last PD-4c orphan -- the same overlap the pipelines sweep hit
with python-basics.md and sql-basics.md, and for the same reason: a file only a
second hop can reach is an orphan whether or not the checker has counted it yet.
Every other reference in the four skills was already linked from its SKILL.md.
One readability fix beyond the mechanical strip. metric-view-advisor.md:758
read "The parent skill's patterns shows each piece on its own" once the
backticked `patterns.md` became a bare word; restored to "the parent skill's
pattern library (patterns) shows", matching the phrasing already used at :19.
Backpressure:
PD-5 70 -> 25 repo-wide; streaming 15 -> 0, metric-views 13 -> 0,
lakebase 9 -> 0, ml-training 8 -> 0
PD-4c 1 -> 0 repo-wide -- the finding is now clear
must-fix 394 -> 348
No other finding count moved in either direction.
skills.py validate and test_repo_bundle_is_canonical fail at this commit:
plugins/** is a generated copy of skills/ and is now stale. Regenerated in a
separate self-heal commit so this content diff stays reviewable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4
Signed-off-by: Brian Madden <bpmadde@icloud.com>
`scripts/skills.py generate` after the PD-5 sweep of streaming, metric-views, lakebase, and ml-training. 56 files = the 14 changed source files x 4 provider bundles. No hand edits; `skills.py validate` is green again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4 Signed-off-by: Brian Madden <bpmadde@icloud.com>
25 links across 15 reference files -- databricks-execution-compute 6, databricks-iceberg 6, databricks-apps-python 5, databricks-lakeflow-connect 2, databricks-serverless-migration 2, databricks-zerobus-ingest 2, spark-python-data-source 2. This clears PD-5 repo-wide. Each link becomes the target's plain name -- basename minus .md, keeping any #anchor. Fenced code untouched (D1). The `../SKILL.md` link sharing a table cell at 4-ingestion-decision-tree.md:22 is a self-parent reference, exempt under D8, and is left as a link. Every reference file in the seven skills was already linked from its SKILL.md, so no routing changes were needed and PD-4c stays at 0. One readability fix beyond the mechanical strip, the largest of the branch. All six databricks-execution-compute links were of the form "Switch to **[Serverless Job](2-serverless-job.md)** when:", where the label is the *compute mode's* name, not a description of the file. Stripping left "Switch to **2-serverless-job** when:", which demotes the three product names this skill exists to choose between into filenames -- the failure the pipelines sweep found at streaming-patterns.md:94. Restored as "Switch to **Serverless Job** (2-serverless-job) when:" in all six, keeping the mode name emphasised and the pointer parenthesised. Backpressure: PD-5 25 -> 0 repo-wide. 228 -> 0 across the branch and its parent. must-fix 348 -> 323 No other finding count moved in either direction. skills.py validate and test_repo_bundle_is_canonical fail at this commit: plugins/** is a generated copy of skills/ and is now stale. Regenerated in a separate self-heal commit so this content diff stays reviewable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4 Signed-off-by: Brian Madden <bpmadde@icloud.com>
`scripts/skills.py generate` after the final PD-5 sweep. 60 files = the 15 changed files under `skills/` x 4 provider bundles. The two changed files under `experimental/spark-python-data-source` have no bundle copy -- the bundle ships the 30 stable skills only. No hand edits; `skills.py validate` is green again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4 Signed-off-by: Brian Madden <bpmadde@icloud.com>
PD-5 and PD-5b are at 0 and PD-4c fell out with them, so the plan's live count table, landing order, and three dependent sections were stale. - Count table restated from a fresh unscoped run: PD-5 0, PD-5b 0, PD-4c 0, SPEC-10a-intra 19 (was 21), NEW-B 0, must-fix 323 (was 475 at branch start). Two rows were already wrong before this branch and are corrected here: SPEC-10a-prose read 3 where the checker reports 0 since `2b9807a` moved those 3 to `SPEC-10a-self-parent` under D8, and that advisory row was missing entirely. `specs/02` had been corrected in `bc9ddb7`; this file had not. - Item 3 replaces the per-skill work list with the finished record and keeps the sweep recipe, now validated across all 15 skills. Two steps are new: the substitution must mirror `strip_fences` rather than re-derive it, and the read-in-context pass now names all four product-name demotions it caught plus the grammar break a de-backticked filename causes. - Item 3a records what a flatten actually costs: three sets of links move with the files -- SKILL.md, inbound cross-skill paths that dangle after the move, and outbound paths whose depth changes -- and the rename must precede the strip, because the strip resolves targets to decide what to rewrite. - Item 14 (PD-4c) and landing-order step 8 are marked absorbed. The branch was never needed: an orphan and a second hop are the same defect, so the sweep that removes the hop must re-route in the same commit. - Step 7b's branch name corrected to `ralph/pd-5-rest`, and the stale `references/appkit/` paths in items 4 and 15 repointed. The `../../../` form the D11 breakdown cited no longer exists in the corpus -- the flatten was its only source -- so that count is re-measured as 17/2/0. AGENTS.md gains one line: `audit_check.py` has no `--root`; measuring another commit means `git worktree add` and running that tree's copy of the script. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4 Signed-off-by: Brian Madden <bpmadde@icloud.com>
GEN-1 asked whether the branch had touched a generated path. `skills.py validate` asks whether those paths are stale. Nothing could satisfy both, so every content sweep landed as a sweep commit (clean here, stale there) plus a self-heal commit (clean there, dirty here). GEN-1 now asks the same question `validate` does: does this tree match a fresh `scripts/skills.py generate`? - `check_gen_1` replaces `check_generated_artifacts`, which was unreachable in any case -- it was defined below `if __name__ == "__main__": main()`, was not in FINDINGS, and raised where the module's house rule is `list[str]`. The new one is registered as must-fix, returns violations, and consults no git. - It delegates to the generator's own drift checks in `generate_all`'s order (codex metadata, plugins, routing, hooks, manifest, bundle) rather than re-deriving them, so GEN-1 and `validate` cannot disagree about what "generated" means. Only `_check_manifest_freshness` is new here, because `validate_manifest` prints and returns bool. - Unit is files out of date; D14 records it. Registered between DESC-1 and TOK-5; repo total stays 568. Verified on `ralph/pd-5-rest`: all four self-heal commits (1df99fa, 7bf0327, 569a135, b34c1b9) and the pipelines one (70101ec) read GEN-1 0, and the five sweep commits that left the bundle stale read 96/36/125/56/60. The old form inverted that and, being merge-base cumulative, climbed 132 -> 317 across the branch: no commit after the first regeneration could ever pass. A source edit mirrored into all four provider trees by hand passes only when it is byte-identical to generator output -- at which point it is a regeneration. AGENTS.md gains two facts that cost real time. `manifest.json` is generated alongside `plugins/` and has to be staged with it: `generate` writes the file but only `git add` puts it in the commit CI validates, so a green working tree is not a green commit. And substituting a plain name for a link is not mechanical -- check whether the label named the file or named a thing. Ten product-name demotions across four sites in the PD-5 sweep were caught by reading diffs; no gate counts them. specs/02 records that PD-4c is a consequence of PD-5, not an independent class. All three orphans cleared inside the PD-5 sweeps: a reference reachable only through a second hop is an orphan, and PD-5's fix already ends "ensure every reference is linked directly from its SKILL.md with a load condition" -- that clause is the PD-4c fix. The row stays to catch a new orphan; the branch does not, so `ralph/pd-4c-orphans` is retired. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0145yTowF7khjiVQLuBkA3T4 Signed-off-by: Brian Madden <bpmadde@icloud.com>
Cherry-picked from 32c297d onto the PD-5/PD-5b line to build ralph/phase1-base, the single branch that carries every completed Phase 1 sweep. Before this, the loop had two divergent heads -- ralph/spec-10a-intra (cross + intra) and ralph/pd-5-rest (PD-5 + PD-5b) -- so no branch held all the work and no two finding counts were comparable. Skills install as subsets of the corpus, so a link reaching into a sibling skill directory dangles by construction. There is no valid cross-skill link form; each becomes prose naming the sibling skill in backticks plus the word "skill", with the sibling's path dropped entirely rather than rewritten. Conflict resolution was semantic, not textual. Six source files had been edited by both sweeps; on every one the two fixes are independent and both were kept: the cross fix drops the sibling path and names the skill, the PD-5 fix strips an intra-references/ link to a plain name. skills/databricks-lakebase/references/{connectivity,off-platform,pgvector}.md skills/databricks-model-serving/SKILL.md skills/databricks-pipelines/references/real-time-mode.md skills/databricks-spark-structured-streaming/references/lakebase-sink-python.md One modify/delete: 32c297d edited references/appkit/lakebase.md, which the PD-5b flatten had already renamed to references/appkit-lakebase.md. The two cross-skill fixes were reapplied at the new path and the stale path dropped. plugins/** regenerated with scripts/skills.py generate and staged here. Backpressure on the integrated base: audit_check.py --only SPEC-10a-cross,NEW-A -> 0 audit_check.py -> must-fix 227 skills.py validate -> 0 Rows still open: PD-6 125, SPEC-10b 64, SPEC-10a-intra 19, DESC-1 6, NEW-C 4, PD-4a 3, PD-1 3, PD-2 3. Blocked unchanged at PD-4b 19 / TOK-5 21 / COMPAT-1 7. Signed-off-by: Brian Madden <bpmadde@icloud.com>
PROMPT_build.md and PROMPT_plan.md are already tracked; these five are the per-branch driver prompts written since. Scaffolding only, no corpus content. Signed-off-by: Brian Madden <bpmadde@icloud.com>
A bare basename is a defect even when the file resolves by luck: the agent is
handed a path that is wrong relative to the skill root, and every reference
that reaches it does so by accident of the reader's cwd.
skills/databricks-mlflow-evaluation/SKILL.md 61 (59 backticked, 2 prose)
skills/databricks-app-design/SKILL.md 3 (backticked)
Zero of the 64 are markdown links, so a ](...)-only regex finds none of them.
Fix applied uniformly: keep the shape, prefix the path.
`patterns-datasets.md` -> `references/patterns-datasets.md`
**Read GOTCHAS.md** -> **Read references/GOTCHAS.md**
Deliberately NOT promoted to markdown link syntax. 52 of the 61 mentions in
mlflow-evaluation sit in the cells of its eight numbered workflow tables --
the best routing in the repo -- and link syntax would have rewritten every
one of those rows to satisfy no gate. The diff is 64 changed lines, each a
pure prefix insertion; everything else in both files is byte-preserved.
databricks-app-design was already using the references/ prefix at lines 14-16,
so this makes that file internally consistent rather than adding a new form.
No product-name demotion to check for here: this class substitutes filename ->
longer path, the opposite direction from the PD-5 sweep's failure mode, and
all 64 labels named files rather than things. Diff read line by line anyway.
plugins/** regenerated with scripts/skills.py generate and staged here;
manifest.json unchanged, as no file path moved.
Backpressure:
audit_check.py --only SPEC-10b -> 0 (was 64)
audit_check.py -> must-fix 163 (was 227); exactly one row
moved, no other row increased
skills.py validate -> 0
unittest discover -s tests -> 149 pass
Signed-off-by: Brian Madden <bpmadde@icloud.com>
Re-landed from e9c265c onto the integrated Phase 1 base. The live population is 19, not the 21 that branch swept: the PD-5b appkit flatten had already cleared the two references/appkit/{jobs,model-serving}.md -> ../platform-guide.md links. All 19 survivors are the same shape -- a markdown link from a references/ file back to its own skill's SKILL.md. There is no '../'-free markdown path from references/x.md to its own SKILL.md: markdown resolves relative to the containing file, so '](SKILL.md)' would point at references/SKILL.md and dangle as NEW-A. The link cannot be rewritten, so it is dropped rather than repaired. references/ loads only after SKILL.md has fired, so a pointer back to it names already-resident content anyway. Convention: backticked `SKILL.md`, with the anchor's section preserved as § "Exact Heading" wherever the anchor carried routing information. databricks-pipelines 5, databricks-ml-training 4, databricks-agent-bricks 3, databricks-ai-functions 3, databricks-aibi-dashboards 1, databricks-lakeflow-connect 1, databricks-metric-views 1, databricks-serverless-migration 1 18 of the 19 sites matched e9c265c byte-for-byte. The 19th (lakeflow-connect/references/4-ingestion-decision-tree.md:22) differed only because the PD-5 sweep had already stripped a neighbouring link on the same line; the SKILL.md substitution itself is identical. All 8 quoted § headings were verified to exist in their target SKILL.md. A mis-quoted heading routes the reader nowhere and no checker row would see it. Four of the 19 labels named a thing rather than the file -- "Overview table", "Widget Index", "CLI Execution", "Failure Reporting Protocol" -- and each survives as prose or as a § clause rather than being demoted to a filename. The 10 remaining '](../' links in the tree are all in experimental/README.md, which sits outside every skill directory and is out of corpus scope. plugins/** regenerated with scripts/skills.py generate and staged here. Backpressure: audit_check.py --only SPEC-10a-intra -> 0 (was 19) audit_check.py -> must-fix 144 (was 163); exactly one row moved, no other row increased skills.py validate -> 0 unittest discover -s tests -> 149 pass Signed-off-by: Brian Madden <bpmadde@icloud.com>
The exemption for the 3 prose `../SKILL.md` occurrences justified itself with
"no subset install can break it -- which is the sole rationale for the SPEC-10a
class". Read literally that also exempts the 19 SPEC-10a-intra links: identical
path, identical kind of source file, yet one row is advisory and the other
must-fix. A spec that contradicts itself over the same string will eventually
be "harmonised" by someone, and the only way to harmonise it as written is to
un-gate SPEC-10a-intra.
The over-claim is the word "sole". The rows are separated by form, not by path,
and there are two independent rationales:
1. Subset-install breakage -- governs SPEC-10a-cross only. `../SKILL.md`
never leaves the skill directory, which is what earns the 3 prose
occurrences their exemption.
2. Dead routing -- governs SPEC-10a-intra. A markdown link is an instruction
to load, and references/ is read only after SKILL.md has fired, so a link
back to SKILL.md points at already-resident content. It is dead routing
whether or not it resolves, and it cannot be repaired: there is no
'../'-free markdown path from references/x.md to its own SKILL.md.
A prose mention carries no load instruction, so rationale 2 does not reach it
and the exemption stands. The exemption is scoped to non-link mentions, which
is what the checker already implements.
No count moves -- both rows are 0. Recorded per the standing instruction not to
silently work around a spec inconsistency.
audit_check.py -> must-fix 144, unchanged
skills.py validate -> 0
Signed-off-by: Brian Madden <bpmadde@icloud.com>
… (3 -> 0)
databricks-spark-declarative-pipelines has never existed in this repo. The
skill that owns the product is databricks-pipelines. An agent told to load a
skill by a name nothing resolves gets no routing and no error.
skills/databricks-docs/SKILL.md:23 prose list of skills to load
skills/databricks-docs/SKILL.md:52 "Load ... skill for workflow patterns"
experimental/spark-python-data-source/SKILL.md:145 Related list
The name only is swapped; surrounding prose is byte-preserved. The
databricks-docs terminology settlement is a separate commit on this branch, and
the "DLT pipelines" wording on the experimental line is left alone -- that skill
is outside the terminology pass this branch was scoped to.
plugins/** regenerated with scripts/skills.py generate and staged here.
Backpressure:
audit_check.py --only PD-4a -> 0 (was 3)
audit_check.py -> must-fix 141 (was 144); exactly one row
moved, no other row increased
skills.py validate -> 0
Signed-off-by: Brian Madden <bpmadde@icloud.com>
The 60-line body named one product three ways -- "Delta Live Tables" at :50,
"DLT" at :53, "DLT / Lakeflow" at :65 -- so a reader could not tell whether
these were one product, two, or three.
Canonical name taken from the skill that owns the product rather than invented
here: databricks-pipelines titles itself "Lakeflow Spark Declarative Pipelines"
and records the alias set at SKILL.md:51 ("SDP = LDP = Lakeflow Declarative
Pipelines = (formerly) DLT").
Settlement: the canonical name throughout, and the legacy names stated exactly
once -- on the Related Skills line, which is the routing pointer, where a
reader who arrived knowing the product as "DLT" needs to recognise it.
:50 Delta Live Tables pipeline -> Lakeflow Spark Declarative Pipelines pipeline
:53 specific DLT features -> specific pipeline features
:65 DLT / Lakeflow pipeline -> Lakeflow Spark Declarative Pipelines
(formerly Delta Live Tables / DLT)
:42 is deliberately untouched. "Data Engineering - Lakeflow, Spark, Delta Lake,
pipelines" enumerates llms.txt documentation categories, not one product under
three names; rewriting it would misdescribe the index this skill exists to
navigate.
Wording only -- no finding count moves, by design.
audit_check.py -> must-fix 141, unchanged
skills.py validate -> 0
Signed-off-by: Brian Madden <bpmadde@icloud.com>
PD-4a 3 -> 0. Also files the two pointers PD-4a's row cannot see: experimental/spark-python-data-source/SKILL.md names databricks-testing and python-dev in its Related list and neither exists. The row hardcodes the single name databricks-spark-declarative-pipelines, so the class is wider than the gate. Intended targets are unknowable, so this is an issue, not a diff. Signed-off-by: Brian Madden <bpmadde@icloud.com>
Reference content sitting beside SKILL.md resolves fine on disk, so nothing
looks broken -- while every references/-scoped check skips it silently: no TOC
check, no orphan check, no one-level-deep check. Three of the four sat in
databricks-core, the repo's entry-point skill.
skills/databricks-core/{databricks-cli-auth,databricks-cli-install,
manual-data-exploration}.md -> references/
experimental/databricks-ai-runtime/docker-images.md -> references/
12 inbound links updated: 11 in databricks-core/SKILL.md (:26, :32, :37, :87,
:141, :149, :150, :151, :156, :157, :158) and 1 in
experimental/databricks-ai-runtime/SKILL.md:33. Each gains the references/
prefix; labels are byte-preserved.
Two pointers outside the skills tree also moved with them:
commands/setup.md:16 -- prose pointer at `databricks-cli-install.md`. This is
hand-written command source, not a generated artifact; the rendered copies
under plugins/*/commands/ follow from generate.
references/manual-data-exploration.md:90 -- linked databricks-cli-auth.md,
which was a root-to-root link before the move and would have become a
reference-to-reference link after it, regressing PD-5 from 0 to 1. Fixed in
the same commit under the established PD-5 shape: keep the name,
parenthesise the pointer. The label named a thing ("CLI Authentication
Guide"), not the file, so the name is kept rather than demoted to a path.
This is regression prevention for a defect this branch would itself have
created, not an out-of-scope PD-5 fix.
PD-6 did NOT rise, contrary to the general expectation for a branch that adds
files to the references population. These four were already inside PD-6's 125:
D10 counts root-level reference files precisely because excluding them is the
blind spot NEW-C describes. The move is population-neutral; the four TOCs land
next and PD-6 falls 125 -> 121.
manifest.json changed here (paths moved) and is staged with the regenerated
plugins/** bundle.
Backpressure:
audit_check.py --only NEW-C -> 0 (was 4)
audit_check.py -> must-fix 137 (was 141); exactly one row
moved, PD-5 held at 0, PD-6 held at 125
skills.py validate -> 0
Signed-off-by: Brian Madden <bpmadde@icloud.com>
…-> 121)
Finishing the files the previous commit moved, not a PD-6 sweep. A reference
this branch relocates ships complete; all four are over 100 lines and carried
no TOC, which is precisely the check that skipped them while they sat at the
skill root.
references/databricks-cli-auth.md 12 entries (H2)
references/databricks-cli-install.md 12 entries (H2 + H3)
references/manual-data-exploration.md 12 entries (H2)
references/docker-images.md 19 entries (H2 + H3)
Derived from the headings already present -- no heading was added, renamed, or
reordered, and no body text changed. Depth is per-file: H2-only where the file
has 12 H2s already, H2+H3 where the H2 count is 6 and the real structure lives
one level down.
House style copied from the 12 references that already pass D8:
`## Contents`, a bulleted list of anchor links, then `---`
(skills/databricks-jobs/references/task-types.md is the exemplar).
Headings inside fenced blocks were masked before parsing -- these files are
dense with bash, and a '# comment' line would otherwise have become a phantom
TOC entry pointing at nothing.
All 57 generated anchors were verified to resolve against a GitHub-style
anchor derivation of the real headings. 0 unresolved. A TOC that satisfies the
checker while linking nowhere would pass the gate and fail the reader.
PD-6 movement is fully attributable: 125 -> 121, exactly the four files this
branch moved. The move commit itself was population-neutral because D10 already
counted root-level reference files.
Backpressure:
audit_check.py -> must-fix 133 (was 137); PD-6 125 -> 121,
no other row moved
audit_check.py --only NEW-C -> 0
skills.py validate -> 0
unittest discover -s tests -> 149 pass
Signed-off-by: Brian Madden <bpmadde@icloud.com>
4 -> 0; PD-6 125 -> 121. Notes two inbound pointers the plan had not recorded (commands/setup.md and the manual-data-exploration -> cli-auth link that would have regressed PD-5), and that the move itself was PD-6 population-neutral because D10 already counts root-level reference files. Signed-off-by: Brian Madden <bpmadde@icloud.com>
The description is the only lever on auto-invocation -- body edits cannot fix a
trigger miss. All six stated capability and named no condition for firing, so
each could only be reached by a user who already knew the skill existed.
Shape applied uniformly, taken from the databricks-data-discovery model:
what the skill does, then "Use this skill when the user asks to '<literal
phrasing>'", then "Do not use for ... (<named sibling>)". The quoted phrasings
are what a user types, not paraphrases of the body.
skill chars yields to
databricks-agent-bricks 122 -> 495 genie-agents, ml-training
databricks-vector-search 133 -> 488 lakebase (pgvector), ai-functions
databricks-execution-compute 173 -> 497 dabs (databricks.yml), jobs
databricks-unstructured-pdf-generation 253 -> 481 synthetic-data-gen
databricks-genie-agents 350 -> 498 data-discovery (Genie One)
databricks-ai-functions 415 -> 476 model-serving, vector-search
Every one names a sibling that could plausibly claim the same request, which is
what makes the negative clause load-bearing rather than decorative.
databricks-genie-agents already had its negative clause and still failed: it
had no positive self-trigger, which is why length alone was never the gate --
at 350 chars it already sat inside the target band.
All six land inside the 300-500 target band (476-498), far under the 1,024 cap.
Four first drafts came in at 504-581 and were tightened rather than left long.
databricks-ai-functions keeps all 12 function names. ai_extract,
ai_parse_document and the rest are literal tokens a user types, so they are
trigger content; the surrounding prose was cut to make room instead.
Resident set against the 2,975-token stable baseline:
stable 2,975 -> 3,347 (+372 tokens, +12.5%)
all-in 3,199 -> 3,571 (+372 tokens)
All six are stable skills, so both figures move by the same amount. 1,477 added
characters / 4 predicts +369; the 3-token gap is D6's single division over the
summed corpus rather than per-skill rounding.
Backpressure:
audit_check.py --only DESC-1 -> 0 (was 6)
audit_check.py -> must-fix 127 (was 133); exactly one row
moved, no other row increased
skills.py validate -> 0
unittest discover -s tests -> 149 pass
Signed-off-by: Brian Madden <bpmadde@icloud.com>
Level 2 loads in full whenever a skill fires, so a SKILL.md over either ceiling
taxes every invocation. The two ceilings bind independently: python-sdk breached
lines only, pipelines tokens only, the other two both.
skill before after
serverless-migration 839 L / 15,683 T 262 L / 4,553 T
python-sdk 625 L / 4,451 T 325 L / 2,494 T
aibi-dashboards 525 L / 8,105 T 311 L / 4,734 T
pipelines 264 L / 8,561 T 197 L / 4,495 T
Content moved verbatim -- extracted by line range and written unmodified. New
references carry an H1 and a one-line read condition; nothing was summarised,
reordered, or rewritten.
serverless-migration references/quick-fixes.md 13,673 ch
references/analysis-output-examples.md 18,000 ch
references/testing-strategy.md 4,966 ch
references/failure-reporting.md (append) 9,175 ch
python-sdk references/core-api-reference.md 8,164 ch
aibi-dashboards references/7-implementation-guidelines.md 9,331 ch
references/6-theme-and-color.md 4,736 ch
pipelines references/api-reference.md 15,543 ch
references/common-issues.md 2,446 ch
Three things had to stay resident and shaped where the cuts fell:
- pipelines "### Legacy DLT Syntax" is the anchor target for four references
that say `SKILL.md` § "Legacy DLT Syntax" after the SPEC-10a-intra sweep.
The API block was cut around it, not through it.
- pipelines "## Common Traps" is one of only two genuine gotchas sections in
the repo and is what earns the skill its PD-8 credit. Moving it would have
raised an advisory row.
- aibi-dashboards "## Widget Index" is the anchor target for
references/4-examples.md:28.
Three regressions this branch created, and fixed in the same commit:
PD-5 0 -> 82 moved content carried `references/x.md` links that became
second hops once the content sat inside references/. Stripped
under the recorded sweep recipe: basename minus .md, anchor
kept, fenced code masked. Every substituted line was read --
the labels were document titles and format names, not product
names, so no name was demoted to a filename.
NEW-A 0 -> 82 the same 82 links, also dangling (references/references/x.md).
One fix closes both rows.
PD-4c 0 -> 9 the moved API tables WERE the routing for nine per-(feature,
language) references. Replaced the dead
"[API Reference tables above](#api-reference)" pointer with a
direct listing from SKILL.md, each with a read condition.
Per-skill character parity, measured on this commit before any TOC lands:
serverless-migration 207,999 -> 209,199 +0.58%
python-sdk 32,829 -> 33,329 +1.52%
aibi-dashboards 124,048 -> 124,994 +0.76%
pipelines 172,458 -> 172,361 -0.06%
All within 2%, which is what shows the content moved rather than vanished.
PD-6 rises 121 -> 125, fully attributable: four of the nine new/extended files
are over 100 lines and TOC-less (quick-fixes 237 L, analysis-output-examples
151 L, 7-implementation-guidelines 176 L, core-api-reference 313 L). The other
five are under 100 lines and exempt; failure-reporting.md was already inside
PD-6's count before this branch appended to it. Their TOCs land next and take
PD-6 back to 121.
Backpressure:
audit_check.py --only PD-1,PD-2 -> 0 (was 3 and 3)
audit_check.py -> must-fix 125 (was 127); PD-3 rollup 4 -> 0
skills.py validate -> 0
Signed-off-by: Brian Madden <bpmadde@icloud.com>
…-> 121)
Finishing the files the split commit created, not a PD-6 sweep. Four of the
nine new/extended files are over 100 lines and so owe a TOC:
references/quick-fixes.md 10 entries (H3)
references/analysis-output-examples.md 9 entries (H4 + H3)
references/7-implementation-guidelines.md 6 entries (H3)
references/core-api-reference.md 13 entries (H3)
analysis-output-examples.md needed one structural change first. The moved
content had exactly one heading and eight standalone bold labels
("**Category A: Unsupported APIs**" ... "**Category H: Job-level config**"),
so no TOC was derivable from it -- a Contents section over a single heading
satisfies the checker and helps nobody. The eight labels are promoted to H3.
Only the markup changes; every word is byte-identical, and this is declared
here rather than buried because it is the one place on this branch where moved
content was touched at all.
The leading H2 that duplicates the file's own H1 is skipped in each TOC --
it exists because the section heading moved verbatim with its body.
All 38 anchors verified to resolve against a GitHub-style derivation of the
real headings, with fenced blocks masked first. 0 unresolved.
PD-6 movement over the branch nets to zero: 121 -> 125 on the split, 125 -> 121
here. Every one of the four is a file this branch created.
Backpressure:
audit_check.py -> must-fix 121 (was 127 at branch start);
PD-1, PD-2, PD-3 are the only rows that
moved across the branch
audit_check.py --only PD-1,PD-2 -> 0
skills.py validate -> 0
unittest discover -s tests -> 149 pass
Signed-off-by: Brian Madden <bpmadde@icloud.com>
4 -> 0. Records the three resident anchors that shaped the cuts (a section targeted by four SPEC-10a-intra pointers, the PD-8 gotchas section, and a widget-index anchor), the three self-inflicted regressions and their fixes, the 2% parity measured on the split commit alone, and PD-6 netting to zero. Signed-off-by: Brian Madden <bpmadde@icloud.com>
Starting with metric-view-advisor.md, the largest file in the repo (822 lines /
59,852 bytes) and the file the plan names as the place to begin. A reader who
opens it with head -100 sees 12% of it and no map of the rest.
references/metric-view-advisor.md 25 entries (H2 + H3)
references/patterns.md 26 entries (H2 + H3)
references/yaml-reference.md 17 entries (H2 + H3)
Derived entirely from headings already present -- no heading added, renamed, or
reordered, and no body text touched. The TOC is inserted after the H1 and its
intro, before the first section heading, matching the house style already in
skills/databricks-jobs/references/task-types.md: `## Contents`, anchor list,
`---`.
Generation rules that matter, and are reused for the rest of the sweep:
- Fenced blocks are masked before parsing. These files are dense with SQL and
bash, and a `# comment` line would otherwise become a phantom entry.
- The H1 is the document title, never an entry.
- Depth is per-file: H2+H3 when the file has <= 12 H2s and <= 45 sections,
H2-only otherwise, so a TOC never buries the file it indexes.
- Anchors are deduped the way GitHub does it (-1, -2 suffixes), computed over
every heading including the H1 so a title/section collision cannot alias.
All 68 anchors verified to resolve against the real headings. 0 unresolved -- a
TOC that satisfies the checker while linking nowhere passes the gate and fails
the reader.
audit_check.py --only PD-6 -> 118 (was 121)
skills.py validate -> 0
Signed-off-by: Brian Madden <bpmadde@icloud.com>
databricks-apps 14 files databricks-pipelines 13 files databricks-spark-structured-streaming 11 files databricks-serverless-migration 8 files 46 files, 710 entries, all derived from headings already present. No heading added, renamed, or reordered; no body text touched. Same generation rules as the metric-views commit -- fences masked, H1 excluded as the document title, per-file H2-vs-H2+H3 depth, GitHub-style anchor dedup. All 710 anchors verified to resolve against the real headings. 0 unresolved. audit_check.py --only PD-6 -> 72 (was 118) skills.py validate -> 0 Signed-off-by: Brian Madden <bpmadde@icloud.com>
spark-python-data-source 8, databricks-lakebase 7, databricks-unity-catalog 7, databricks-aibi-dashboards 5, databricks-apps-python 5, databricks-iceberg 5, databricks-mlflow-evaluation 5, databricks-zerobus-ingest 5 47 files, 601 entries, derived from headings already present. Includes databricks-unity-catalog/references/5-system-tables.md (1,043 lines) and databricks-aibi-dashboards/references/4-examples.md (1,021 lines), the two longest files in the repo after metric-view-advisor. All 601 anchors verified to resolve. 0 unresolved. audit_check.py --only PD-6 -> 26 (was 72) skills.py validate -> 0 Signed-off-by: Brian Madden <bpmadde@icloud.com>
databricks-ml-training 4, databricks-vector-search 4, databricks-dabs 3,
databricks-dbsql 3, databricks-lakeflow-connect 3,
databricks-execution-compute 2, databricks-synthetic-data-gen 2,
databricks-agent-bricks 1, databricks-ai-functions 1,
databricks-model-serving 1, databricks-python-sdk 1,
spark-python-data-source 1
26 files, 324 entries, derived from headings already present.
Two files needed handling the generator could not do mechanically:
experimental/spark-python-data-source/references/implementation-template.md
is one 115-line fenced template with a single section heading at line 129.
The generated TOC landed there -- past D8's 60-line window -- so the file
stayed in the population and took a second TOC on the next pass. The
duplicate is removed and one TOC sits after the intro, where a reader looks.
Its single entry is the file's real structure; a line naming the four modes
the template covers does the orienting work a one-entry list cannot.
Two TOC labels duplicated a "(Beta)" marker from their own heading, which
raised blocked row TOK-5 from 21 to 23 -- iceberg/1-managed-iceberg-tables.md
and lakebase/synced-tables.md. The marker is stripped from the LABEL only;
the anchor still derives from the full heading, so the link resolves and the
heading keeps its marker. TOK-5 is back to 21. A blocked row must not move on
a sweep that is not about it, and an index duplicating a marker is not the
uncontained marker that row exists to count.
Corpus-wide re-verification after every batch: 2,034 in-file anchors across 213
markdown files, 0 unresolved.
PD-6 is the last must-fix class.
audit_check.py --only PD-6 -> 0 (was 121 at branch start, 125 at audit)
audit_check.py -> must-fix total 0; blocked rows unchanged at
PD-4b 19 / TOK-5 21 / COMPAT-1 7; no advisory
row moved
skills.py validate -> 0
unittest discover -s tests -> 149 pass
Signed-off-by: Brian Madden <bpmadde@icloud.com>
…facts PD-6 121 -> 0, and must-fix total reaches 0. The plan gains a Phase 1 closure summary with the branch chain and final counts. AGENTS.md gains the two operational facts this sweep cost a re-run to learn: a sweep that creates or moves a reference file regresses PD-5/NEW-A/PD-4c and can raise TOK-5, none of which --only can see; and the TOC heuristic's 60-line window is a placement constraint, not just a detector. Signed-off-by: Brian Madden <bpmadde@icloud.com>
Signed-off-by: Brian Madden <bpmadde@icloud.com>
bpmadde
requested review from
a team,
camielstee-db,
dustinvannoy-db,
lennartkats-db and
simonfaltum
as code owners
August 2, 2026 08:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Structural remediation across all 32 skills: reference-path, progressive-disclosure, and description-trigger defects. No prose rewritten, no guidance changed.
857 files changed reads worse than it is. 662 are
plugins/**andmanifest.json—scripts/skills.py generateoutput, regenerated in-commit so the four provider mirrors stay in sync with source. 193 are human-authored: 191 skill files plus one checker and its test.PRs cannot be merged here, so this is offered for hand-migration. Each finding class is a contiguous commit range and can be taken or dropped independently.
Numbers
Measured by
scripts/audit_check.py, included in this PR. must-fix at50ccd08(this base): 568. At HEAD: 0.Three rows are excluded as blocked on maintainer decisions; they are unchanged and filed as issues. The checker sums
must-fixseverity only.Reproduce:
git worktree add /tmp/base 50ccd08 && cp scripts/audit_check.py /tmp/base/scripts/ && cd /tmp/base && python scripts/audit_check.py --detailsFinding classes
references/—4fcb43e70101ec70fd7941df99fa6fe70867bf0327c255eed569a135c80b92eb34c1b9../traversals, 90 to 0 (NEW-A 6 to 0) —96530e71540a88../links, 19 to 0 —5b8b13b40dc020883a130f6b53a674ae428a60ade03c7ace4efcee8c395c94961e0817f56b7c80289e5d74ae428andefcee8ccarry the same125 -> 121in their subjects. That is a message error in the second. NEW-C moved four files into the PD-6 population and TOC-ed them; PD-1 created four more and TOC-ed those. Each nets zero. PD-6 stood at 121 when its own sweep opened.PD-4c (orphan reference files, 3) is inside the 568 and went to zero as a side effect of PD-5 — once reference-to-reference links were flattened, the orphans had inbound routing. No branch of its own for that reason.
The checker
scripts/audit_check.pymeasures every row above.--only IDscopes a run; exit status is the gate. Scansskills/andexperimental/only. Adopting it as CI is optional and separable from the fixes.Three limits, stated up front:
(absent)sorts ahead ofRequires..., so the row only reaches zero if the surviving shape is nocompatibilityfield at all. Fix is a configured target, not a derived one. Left as-is pending the policy decision in the issue.experimental/spark-python-data-source(databricks-testing,python-dev). Recorded in an issue rather than guessed at.GEN-1 reads 0 at
50ccd08— upstream generated tree was already fresh. The guard never had to repair anything; it kept this branch honest.What this cost
DESC-1 raised the always-resident metadata set from 2,975 to 3,347 tokens stable (3,199 to 3,571 all-in), +12.5%. That is the price of giving six skills real trigger conditions instead of bare topic labels, and it is the one measurement this PR makes worse.
Three sections that could not move
databricks-pipelines### Legacy DLT Syntax— anchor target for four section pointers created by the SPEC-10b sweep.databricks-pipelines## Common Traps— earns that skill its PD-8 credit; moving it would have raised an advisory row.databricks-aibi-dashboards## Widget Index— targeted by a reference.Content was moved verbatim in every split, never summarised. Per-skill total character counts land within 2% of pre-split, which demonstrates content moved rather than vanished.
Not included
Three rows unchanged and filed separately: PD-4b (19, core routing completeness), TOK-5 (21, uncontained preview/beta markers), COMPAT-1 (7, compatibility pin shapes). Each needs a maintainer decision about intended shape, not a mechanical fix.
All commits are signed off per DCO.