Generate PyDABs reference doc pages from generated modules - #6727
Merged
Merged
Conversation
The Sphinx .rst pages under python/docs/ were hand-written and listed only 5 resources, while codegen now emits 31 resource modules — so the published Python reference docs (databricks.github.io/cli/python) went stale. Generate the per-resource .rst pages and index.rst toctree in pydabs-codegen, driven by scanning the generated databricks/bundles/*/ package dirs (decoupled from RESOURCE_NAMESPACE). The existing hand-written pages regenerate byte-identical; core.rst stays hand-written. CI's generate-check + git diff now enforces the doc list never drifts from the generated resources. Co-authored-by: Isaac <no-reply@databricks.com>
Extract the RST page and index layout from generated_docs.py into doc_page.rst.tmpl and doc_index.rst.tmpl, loaded via string.Template like the existing *.py.tmpl codegen templates. This lets the doc structure be reviewed and tracked separately from the generation code. Output is byte-identical. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
added this pull request to stack #6730
September 17, 2026 10:08
janniklasrose
left a comment
janniklasrose
left a comment
Member
There was a problem hiding this comment.
tested task pydabs-docs:
/Users/jan.rose/pub/cli.worktrees/pr-6727/python/databricks/bundles/sql_warehouses/_models/spot_instance_policy.py:docstring of databricks.bundles.sql_warehouses._models.spot_instance_policy.SpotInstancePolicy:5: ERROR: Malformed table.
+-------+--------------------------------------+--------------------------------+
| Cloud | COST_OPTIMIZED | RELIABILITY_OPTIMIZED |
+-------+--------------------------------------+--------------------------------+ [docutils]
/Users/jan.rose/pub/cli.worktrees/pr-6727/python/databricks/bundles/sql_warehouses/_models/spot_instance_policy.py:docstring of databricks.bundles.sql_warehouses._models.spot_instance_policy.SpotInstancePolicy:8: WARNING: Blank line required after table. [docutils]
/Users/jan.rose/pub/cli.worktrees/pr-6727/python/databricks/bundles/sql_warehouses/_models/spot_instance_policy.py:docstring of databricks.bundles.sql_warehouses._models.spot_instance_policy.SpotInstancePolicy:9: WARNING: Line block ends without a blank line. [docutils]
/Users/jan.rose/pub/cli.worktrees/pr-6727/python/databricks/bundles/sql_warehouses/_models/spot_instance_policy.py:docstring of databricks.bundles.sql_warehouses._models.spot_instance_policy.SpotInstancePolicy:9: ERROR: Malformed table.
+-------+--------------------------------------+--------------------------------+
| AWS | On Demand Driver with Spot Executors | On Demand Driver and [docutils]
/Users/jan.rose/pub/cli.worktrees/pr-6727/python/databricks/bundles/sql_warehouses/_models/spot_instance_policy.py:docstring of databricks.bundles.sql_warehouses._models.spot_instance_policy.SpotInstancePolicy:11: WARNING: Blank line required after table. [docutils]
/Users/jan.rose/pub/cli.worktrees/pr-6727/python/databricks/bundles/core/_resources.py:docstring of databricks.bundles.core._resources.Resources:23: WARNING: py:meth reference target not found: add_job [ref.meth]
Collaborator
Integration test reportCommit: eb02251
Top 3 slowest tests (at least 2 minutes):
|
Contributor
Author
This is a issue with the comment in the protos, we would have to change the upstream descriptions, or special case this, I would prefer doing the former |
Member
please fix this upstream. but with the auto-gen + pipeline you're building in this stack, we need a way to surface these. erroring is probably not ok, ideally this should fail at SDK bump stage with an override that we can burn down again upstream |
Contributor
Author
Just a clarification this is not an error but a warning so the docs would be generated but they would just be broken |
The sql.SpotInstancePolicy description is a hard-wrapped ASCII grid table that docutils rejects as malformed, so the sql_warehouses page rendered broken and the Sphinx build logged errors. Add a burn-down override list in jsonschema_patch that rewrites such upstream descriptions into valid reStructuredText until the proto comment is fixed upstream (see sqlgateway/scheduler/api/proto/endpoint_common.proto). The override raises if it becomes a no-op, flagging entries to remove once upstream lands. Pairs with the strict (-W) docs build added later in the stack. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
Added an override here, and the hard error is added to the other PR on this stack |
janniklasrose
approved these changes
Sep 18, 2026
deelaka
pushed a commit
to deelaka/databricks-cli
that referenced
this pull request
Sep 23, 2026
…#6729) ## Stack - databricks#6727 — generate the PyDABs `.rst` doc pages from the generated modules (base) - databricks#6729 — publish the built HTML to `gh-pages` + make the docs build strict ← **this PR** ## Changes Two related pieces of the PyDABs docs pipeline: **1. Auto-publish the docs to `gh-pages` on release.** A new `update-pydabs-docs` GitHub Actions workflow rebuilds the PyDABs Python reference docs and publishes the HTML to the `gh-pages` branch on every release tag (and on demand via `workflow_dispatch`), replacing the manual copy that had left https://databricks.github.io/cli/python/ stale (5 pages vs. the 32 the build now emits). Modeled on `update-schema-docs.yml`: - Trigger: `push` tags `v[0-9]+.[0-9]+.[0-9]+*` + `workflow_dispatch`. - Permissions: `contents: write`, `id-token: write` (setup-jfrog GOPROXY exchange). - Checkout `main`, set up JFrog / Go / uv, then `git worktree add` `origin/gh-pages`. - Build with `go tool -modfile=tools/task/go.mod task pydabs-docs`. - Refresh only the `gh-pages` `python/` directory wholesale from `python/docs/_output/`, so a dropped resource loses its page; top-level `.nojekyll`, `index.html`, and `experimental/` are untouched. - Commit as `github-actions[bot]` and push directly to `gh-pages`, no-op when there is no diff. **2. Make the docs build strict (`-W`).** Add `-W` to the `pydabs-docs` sphinx-build command so warnings — e.g. malformed upstream descriptions or unresolved cross-references introduced by an SDK bump — fail the build instead of passing silently. This makes both the existing `python_docs` CI job (every PR, including SDK bumps) and the new publish workflow strict, surfacing docs breakage at bump time. Fixes the one pre-existing warning this exposes: the `Resources` docstring referenced `add_job` with `:meth:`, but `add_job` is a generated method Sphinx cannot resolve — changed to an inline literal. Stacked on top of databricks#6727, so the strict build already includes its generated pages and the override for malformed upstream descriptions. ## Tests From repo root, `./task pydabs-codegen` then `./task pydabs-docs` both exit 0 under `-W` (the `add_job` fix + databricks#6727's override clear the only warnings). YAML validated locally. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: 04741e9
192 interesting tests: 135 FAIL, 57 MISS
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
.rstdoc pages from the generated modules (base) ← this PRgh-pages+ make the docs build strictWhy
The PyDABs Python reference docs (https://databricks.github.io/cli/python/) went stale. Codegen now emits 31 resource modules under
python/databricks/bundles/*/, but the Sphinx.rstpages were hand-written and listed only 5 (core,jobs,pipelines,schemas,volumes).What
Generate the per-resource
.rstpages and theindex.rsttoctree insidepydabs-codegen:python/codegen/codegen/generated_docs.py—write_docs()scans the generateddatabricks/bundles/*/package dirs (decoupled fromRESOURCE_NAMESPACE, robust to future renames), writes one page per resource, and regenerates theindex.rsttoctree. Called frommain.pyafter the existing generation.core.rststays hand-written. The 4 previously hand-written resource pages (jobs,pipelines,schemas,volumes) regenerate byte-identical — no diff.index.rst's prose header is preserved verbatim; only its toctree grows.SQL Warehouses,MCP Services) handled via a small override map.doc_page.rst.tmplanddoc_index.rst.tmpl(loaded viastring.Template, like the existing*.py.tmplcodegen templates) so it can be reviewed separately from the generation code.python/Taskfile.yml: added the generated doc outputs to the task'sgenerates:list.Also fold in a related fix surfaced in review: some upstream API descriptions aren't valid reStructuredText and break the Sphinx build (e.g.
sql.SpotInstancePolicy, whose comment is a hard-wrapped ASCII grid table docutils rejects as malformed).jsonschema_patch.override_descriptionsrewrites such descriptions into valid RST until the proto comment is fixed upstream (seesqlgateway/scheduler/api/proto/endpoint_common.proto). The override raises if it becomes a no-op, flagging entries to remove once upstream lands.-Wdocs gate that surfaces any new such breakage (at PR/SDK-bump time) lands in Auto-publish PyDABs reference docs to gh-pages on release #6729.CI's
generate-check+git diff --exit-codenow enforces the documented resource list can never drift from the generated modules.Testing
./task pydabs-codegen— regenerates code + 31.rstpages +index.rst; the 4 existing pages show no diff; idempotent on re-run../task pydabs-docs— Sphinx build succeeds and now emits all 32databricks.bundles.*.htmlpages; thesql_warehousespage renders a proper table instead of the malformed one.This pull request and its description were written by Isaac.