🐛 An incremental build re-writes every page showing a need that changed by chrisjsewell · Pull Request #2414 · useblocks/sphinx-needs

chrisjsewell · 2026-10-30T11:10:48Z

What

packages/sphinx-needs (new internal module _incremental.py, _-prefixed like the package's other new internal modules).

The needs each re-read document gives are compared, as read, with what it gave before: an env-purge-doc stash and env-get-updated,
before the environment is pickled, so post-processing still happens after the pickle. On a change, Sphinx re-writes (never re-reads):

every view page (needtable, needlist, needflow, needgantt, needsequence, needpie, needbar, needextract, needuml/needarch, needreport, need_count, ndf);
the pages of the affected needs: changed, named by a changed id-needextend, or computing a value from an affected need (a dynamic function read by its arguments,
whether written or set by a needextend; a derive rule over its links, needextend-added ones included, its back links or from), to a fixpoint;
the pages of their link targets, old and new, and of the needs linking to them;
every page whose need role names one.

What a computed value reads is decided by an allow-list: a value counts as reading only its own need when every argument is statically known
to read that need's own fields; anything else (variant conditions, functions of your own, opaque filters, any field other needs decide, such as back
links, dead-link flags or constraint results) counts as reading every need. Every core field is classified, and a test fails on an unclassified new one.
A need that only moved (lines added above it) re-writes the views and the pages whose roles name it, and only when nothing anywhere reads a line number;
prose below the needs re-writes nothing more. Projects with links computed to needs not named before resolution (a dynamic function in a link field or a
needextend's links, a links rule, content_links with from) re-write every need-relevant page on each change, as does a changed filter-needextend.
The rules err toward re-writing a page, never toward a stale one.

Views without a sort now list needs in document order on every build (an incremental build used to list re-read needs last). ENV_DATA_VERSION is 16
(master's 14 is the hops on a need's source, #2399; its 15 is c.is_a, #2401).
Other open branches also claim 16 (#2398's fix, the source-hops slice 3a): whichever lands first keeps it, each later one renumbers at its
merge of master and keeps every version paragraph.
Docs: a page on incremental builds, saying exactly which pages are re-written and when the fallbacks fire.

Master's newer readers, checked against the rules

Merged with master after #2383–#2406. Each new reader is covered, each by a scratch-vs-incremental case (or, for schema findings, a test of the incremental build's findings), and where a rule of this PR carries it, a mutation of that rule turns the case red:

c.is_a('x') (✨ Filters follow inherits: c.is_a('req') selects a type and the types inheriting from it #2401) reads the need's type and the need types' inherits. In a filter of a rule or a call it reads the candidates' types, which the
rules already follow (a type is a field the need's own document gives); a needtable, needextend or any view using it is re-written or re-applied as before.
An inherits edit is a change of the need types, which re-reads every document (reread_on_need_types_change, 🐛 A derive added to or removed from a needs_types entry is not applied on an incremental build (needs_types rebuilds the HTML only) #2287) — also when it is made in
ubproject.toml alone. Case: inherits-changed (toml only: req declares a field with a default, which spec takes once it inherits req, so the
case goes red with that re-read switched off; a needtable and a needextend filtering on c.is_a, a count over links where c.is_a(...) copied by a
need on another page).
Per-type back-link counts and severity (✨ Per-type declarations: a severity, a count of incoming links, and an id pattern #2395) are schema findings, which are validated over every need on every build: removing the one link to a
need on another page reports it on the incremental build exactly as on a fresh one (pinned by a test). Per-type id_regex is matched at read time,
on the need's own document, and is a need-type change too.
Derive where beside copy, empty on any/all (✨ Derive: empty on any/all, where beside a copy, and the derive corpus follow-ups #2384): a where narrows the needs the rule already reads (over links), and empty is a
literal. Case: rule-with-where-and-empty-read (rules declared on one need type alone in ubproject.toml, ✨ Extension need types and fields per type: a needs_types entry narrows an extension's type, add_field(types=…), and sphinx-test-reports' fields on their own types #2400, read two hops away).
Source hops (✨ Every need records where its text came from, as hops on its source (provisional; nothing shown changes) #2399): nothing renders from them (need.source.hops is provisional; the legacy keys derived from them are classified core fields).
They do record a need's lines, so the moved-only comparison now leaves out each hop's start and end line, and the source classes' private line field:
after the merge it crashed (NeedItemSourceDirective.__init__() got an unexpected keyword argument '_lineno') on every build in which a need moved.
Cases: prose-above-included-need (a need written in an included file, which its hops record), prose-above-imported-need (an imported need,
whose line is kept beside its hops) and need-moved-to-another-included-file (the same text from another file is a changed need), each with the
exact set of pages it writes.
create_need (✨ create_need: a need's fields and links as two mappings, the need returned — and a field named like an add_need parameter no longer ends the build #2406): its needs and sources are compared as any other; it adds no reader. A source an extension passes that is not a dataclass
and has no value __eq__ compares by identity, so its need counts as changed on every re-read of its document: pages re-written, never stale.
Extension types and fields per type (✨ Extension need types and fields per type: a needs_types entry narrows an extension's type, add_field(types=…), and sphinx-test-reports' fields on their own types #2400): declared by the configuration (a need-type change re-reads every document) and read through the
schema the rules already ask (rule_for(type)); a type-local derived field is covered by the case above.

For the source-hops work

The moved-only comparison (_without_lines in need_item.py) reads the source classes' private _lineno and rebuilds #2399's provisional hop
dataclasses from outside _need_source.py: a new field there fails safe (a moved need counts as changed), a custom __init__ on a hop class would
raise (the incremental suite goes red); the clean seam is a method on HopsSource / NeedSourceHop giving that key, left to that work.
The source-hops slice 3a (one frozen NeedItemSource in place of these classes) was merged onto this branch in a scratch clone by that work's
reviewer: _without_lines works unchanged there and these tests pass; whichever of the two lands second moves the comment to the new seam.

Why

Fixes #2376, fixes #346. Related: #442.

Editing a need's title and rebuilding incrementally re-wrote only the edited page: every needtable/needflow page, every :need: role, the need's link
targets (back links) and the needs linking to it kept the old values until a full build, as did the needs a needextend changed or stopped changing.

Not covered: a changed needs_external_needs file (#353), and moving a need between two documents in one incremental build (pre-existing, identical
before this change; #2391).
A :need_outgoing: / :need_incoming: naming a need that does not exist ends even a fresh build (#2413); on master an incremental build left
that page stale and green, with this change it is re-written and fails the way a fresh build does.

Tests

A scratch-vs-incremental equivalence oracle: every page on disk after the incremental build equals a fresh build's, over the fixture's scenarios
(several also at -j 2), plus precise-set tests. Two independent reviews and four validation rounds probed 34 edit shapes; every hole they found
is now a test, and every dependency rule fails a test when mutated. After the merge with master: six more scenarios and four tests for master's readers (51 tests in the module).
Full builds at base and head are identical for 14 projects (html and needs builders), apart from the env version in searchindex.js.
Cost on a 1000-need / 211-page project: a title edit writes 14 pages in about 1.5–2 s (before: 2 pages, the views stale; re-writing every need page:
211 pages, 22 s); with a field default [[copy('title')]] the same.
uv run poe lint 0, uv run poe typecheck 0, full uv run poe test-needs 4119 passed / 15 skipped (all 4134 collected), uv run poe smoke-needs 0.
After the module's rename to _incremental.py: lint, typecheck and smoke-needs 0 again, and the incremental, need-types and need-source modules 103 passed.

Checklist

I wrote this change myself and have read every line of it; it was not generated automatically from an issue.
I ran the package's tests (uv run poe test-needs) and they pass.
Documentation is updated where behaviour or options change.
The package's docs/changelog.rst has an entry under Unreleased.
uv run poe lint and uv run poe typecheck pass.

A page whose needtable, needflow or other view lists a need, whose role names it, or whose needs link to it (or are linked from it) kept the need's old values on an incremental build until a full one (#2376). The needs of every re-read document are now compared, as read, with what that document gave before, and on a change the pages showing it are re-written (not re-read): every view, the pages of the needs affected (changed, named by a changed needextend, or computing a value from one through a dynamic function or a derive rule), of their link targets and of the needs linking to them, and those whose need roles name one. A changed needextend with a filter, or links computed to needs that are not named, re-write every need-relevant page (#346). An edit of prose only re-writes nothing more. The needs are ordered by document before they are resolved, so a view without a sort lists them as a fresh build does. ENV_DATA_VERSION is 14: the categories and role references are recorded when a document is read.

Every rule of the fix is now the only one re-writing some page: a need linking to the edited one on a page of its own (with the link titles shown), a rule summing a rule's value on another, the dynamic function on a page of its own, the extends naming needs no other rule reaches, and a content reference to a need of another page. Removing any one rule fails a case.

A page under Performance names every page an incremental build re-writes when a need changes, what counts as a change, the changes that re-write every need-relevant page, and that external needs are not compared (#353). The changelog's Unreleased section carries the fix (#2376, #346) and the ENV_DATA_VERSION bump to 14.

…er need What an incremental build's dependency check reads of each need type's derive rules depends on the type alone, so it is worked out the first time a type is met rather than for every need.

…argument, moved needs Review fixes for the incremental re-write of the pages showing a changed need: - A needextend's links reach the dependency graph whole: the extends as they were count as well as those as they are, so a link one no longer adds leaves its old target's page, and a derive rule over links reads the links an extend adds. - A value a needextend sets with a dynamic function is computed on its need: an extend by id makes its need read what the call reads; one by filter whose call reads needs it does not name re-writes every need-relevant page. - A built-in dynamic function is read by its arguments: ``copy`` of the need's own field reads no other need, ``copy`` with a need_id that need, and ``check_linked_values`` or a ``links_only`` ``calc_sum`` the needs it links to; a function of your own, or a filter, may still read any need. - A need that only moved (an edit above it, its line numbers alone differ) re-writes the views alone, and an extend that only moved re-writes nothing. - The purge no longer scans every need for every document: a document new to the environment gave nothing, and the needs are indexed by document once. The fixture gains the scenarios each of these rules is the only one to catch: extend links removed and edited at either end, an extend-added link read by a derive chain, an extend call by id and by filter, an extend with dynamic links, a filter extend with literal links, a removed link and a removed document, a rule-less need type read first, and the two precise-set tests for an edit above the needs and for a need nothing reads.

…ritten The docs page states the dynamic functions as the build reads them (by their arguments, written or set by a needextend), the links a needextend adds, a need that only moved re-writing the views alone, and exactly when the fallback to every need-relevant page fires: on every change in a project with links computed to needs not named before resolution or with a filter needextend whose call reads needs it does not name, and after a change of a filter needextend or of a need one links to or names. The changelog entry no longer says any prose edit re-writes nothing more: prose above a need moves it.

… value Two refinements of the previous commit assumed what a computed value reads: - A back-link field (``<link type>_back``) is written by the needs linking to the need it is read on. A ``copy`` (or a derive rule's field) of one now depends on those needs: of its own need's, on the needs linking to it; of a named need's, on that need's back links changing; over its links, on its link targets' back links. A filter naming back links reads any need. - A need that only moved re-writes the views alone only when no dynamic function, derive rule or needextend filter reads a line number (``lineno``, ``lineno_content``) — a function of your own, or a filter naming ``lineno``, may; otherwise a moved need is a changed one. The fixture gains a copy of a need's own back links, of a copy of them, and of another need's, each on a page of its own, and a line number read by a dynamic function and by a derive rule; each is red against the previous commit and under its mutation.

…lues The docs page says a back-link field read by copy depends on the needs linking to the need it is read on, that copy with a need.<field> id may read any need, and that a need that only moved is a changed one when a computed value or a needextend filter reads a line number. The changelog names the two filter-needextend fallbacks the docs page has.

…lates read every need The two narrowings of the dependency rules are now allow-lists, so a shape they do not know re-writes a page rather than leaving one stale: - A computed value reads only its own need when it is a built-in call or a derive rule whose arguments are all known to name the need's own fields. A variant (``<<…>>``) or any other item, a call or rule of an unknown shape, a filter reading ``needs`` or ``current_need`` by a computed key, is taken to read every need and every field; a variant in a link field computes links to needs not named before resolution. A ``<{ var.* }>`` reference reads the configuration only. - A need that only moved re-writes the views alone only when nothing in the project may read a line number: no such computed value, needextend filter, link condition or constraint, and neither the :need: role's template nor a layout naming one. The fixture gains a variant reading its need's back links and one reading a line number, each copied by a need on another page, and a role template showing the line number; each is red against the previous commit.

…oward re-writing The docs page says that only a computed value of a known shape is read narrowly, that anything else (a variant, a function of your own, a filter reading needs) is taken to read every need, and when a need that only moved re-writes the views alone: when nothing in the project can read a line number, the :need: role's template and layouts included. The changelog says the same.

… a moved need's roles - One named set, OTHER_WRITTEN_CORE_FIELDS, holds the core fields whose value other needs or the whole set decide when the needs are resolved: the dead-link flags, the constraint results, modifications and is_modified, parts; every <link type>_back is one too. A call, rule or filter reading one reads every need (back links keep their narrower rule). OWN_CORE_FIELDS holds the rest, and a test fails when a core field is in neither set. - A need that only moved also re-writes the pages whose roles name it: a role's text ([[lineno]]) or needs_role_need_template may show its line, so the template is no longer scanned. The fixture gains a dead-link flag read by a call and by a derive rule (the latter stale at the base too), a role text showing a line number, alone and beside a change elsewhere; each is red against the previous commit.

…oved need The docs page names the values computed when the needs are resolved (the dead-link flags, the constraint results, modifications, parts), read as reading every need, and says a need that only moved re-writes the pages whose roles name it as well as the views. The changelog says the same.

ENV_DATA_VERSION becomes 16: master's version 14 (the hops on a need's source) and 15 (c.is_a follows inherits) are kept as they are, and this branch's paragraph (views and need roles recorded at read time) is Version 16. The changelog entries join master's Unreleased Bug fixes.

…ut their lines Since every need records where its text came from as hops on its source, a need written at other lines differs in the start and end line of its hops too, and the source classes keep their line in a private field: the comparison that told a moved need from a changed one rebuilt the source with dataclasses.replace, which the hop-carrying classes refuse (NeedItemSourceDirective takes no '_lineno'), so every incremental build in which a need moved stopped in env-get-updated. The source is now compared field by field, leaving out its line fields and each hop's start and end line.

The scratch-vs-incremental oracle gains a project declaring fields of a need type alone in ubproject.toml, derived with a 'where' beside a 'copy', an 'all' with 'empty', and a count over the links c.is_a('req') selects, read by a need on a page of its own; a needtable and a needextend whose filters use c.is_a. Cases: a value those rules read changes, and 'inherits' changes in ubproject.toml alone. A need written in an included file moves when prose is added above it there, and re-writes the views alone. A need type's count of its back links is reported on the incremental build that removes the link, as on a fresh one.

…told from a change A need with no link re-writes the same pages whether it moved or changed; linked to REQ_002, a change would re-write the page showing its back link as well, so the precise test now tells the two apart.

…, need types, schema counts A where narrowing the needs a rule reads, and c.is_a in it, read those needs alone; a change of the need types, inherits included, re-reads every document; what the schema reports, a type's count of the links it receives included, is checked over every need on every build.

create_need takes a source and its fields and links as mappings; it changes nothing the incremental rules read, and ENV_DATA_VERSION stays 16.

… and what it keeps Prose above a needimport moves its need and no more, though the line of that need is kept beside its hops rather than on them; the same need included from another file is a changed need, since its hops say where it came from.

…ource hops It reads the source classes' private line and rebuilds the provisional hop dataclasses from outside their module: a new field fails safe, a custom __init__ would not, and a method on HopsSource / NeedSourceHop is the seam to move it to.

…n it is read req declares a field with a default in ubproject.toml, which spec takes once it inherits req: the needs of the unchanged specs page then differ only if they are read again, so the case rests on the re-read of every document a need-types change triggers.

codecov · 2026-10-30T11:12:35Z

Codecov Report

❌ Patch coverage is 96.66667% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.83%. Comparing base (8bd3798) to head (6796f89).

Files with missing lines	Patch %	Lines
...ages/sphinx-needs/src/sphinx_needs/_incremental.py	96.58%	12 Missing ⚠️
...ackages/sphinx-needs/src/sphinx_needs/need_item.py	91.30%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2414      +/-   ##
==========================================
+ Coverage   94.78%   94.83%   +0.05%     
==========================================
  Files         145      146       +1     
  Lines       25434    25839     +405     
==========================================
+ Hits        24107    24505     +398     
- Misses       1327     1334       +7

Flag	Coverage Δ
ai-index	`100.00% <ø> (ø)`
codelinks	`96.31% <ø> (ø)`
mounts	`94.27% <ø> (ø)`
pytests	`94.67% <96.66%> (+0.08%)`	⬆️
reports	`97.05% <ø> (ø)`
ub-test-reports	`91.05% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…e package's other new internal modules are

…es 17 #2414 took version 16 for what an incremental build records. The one source class's paragraph becomes version 17 ("versions 14 to 16 stored one of five classes"), both paragraphs kept, and the upgrade fence's LAST_VERSION_OF_THE_FIVE_CLASSES is 16, as its check against data.py requires. need_item.py's imports are the union of both sides'.

… the source, as strict xfails NeedItemSource._without_lines() is to give the key NeedItem.moved_only compares (the source without its line fields and its hops' lines, the key #2414's _without_lines built from outside the class), and NeedSourceHop._without_lines() a hop's: for a directive's, an included need's, a service's, an imported and an API source, a need written lower compares equal and one placed elsewhere does not. Red: the methods do not exist.

… of a need that only moved The source gives the comparison key NeedItem.moved_only uses itself, beside the fields it knows: the class and every field but the line fields and dict_repr, its hops through NeedSourceHop._without_lines() (start and end line left out; _need_source had no helper doing that, so the hop has its own method). _without_lines calls it for a NeedItemSource and keeps #2414's fallback for a third-party source as written; its comment no longer names HopsSource. The key is the one #2414 built from outside (pinned), and #2414's test_incremental_rewrite.py is unchanged and green.

master squash-merged the derive branch this one stacked on (#2384), and has since gained per-type declarations (#2395), extension types (#2400), c.is_a (#2401), the incremental rewrite rules (#2414) and more. Conflicts kept both sides: the changelog's Unreleased bullets; LinkSchema's severity beside summary in the basic-doc snapshot; a path's select default in derive.py; the derive corpus gains master's where-follows-inherits case ahead of this branch's eleven, 103 cases.

…the third-party fallback The comparison key must fail safe, as #2414's did: a NeedItemSource subclass with one more field, two sources differing only in it, must not compare as a need that only moved. Red now: the seam lists its fields by hand (review B's S9). The fallback for a source that is no NeedItemSource, which no test reached since the seam: one that is no dataclass is compared as it is, so a different line is a change; a third-party dataclass is compared by its fields but the line fields, its hops without their lines (B's S3, S4, S8).

… is deprecated in favour of create_need (#2419) An extension now creates a need with one call, giving its location through one stable public class: `create_need(app, state, NeedItemSource.from_directive(self), "req", title, fields={...}, links={...})` in a directive. For a need at a line the caller gives, the location is `NeedItemSource.on_page(docname, lineno=lineno)`: the page and that line, as `add_need` built it (sphinx-codelinks' needs sit at their directive's parser line this way). `NeedItemSource` is importable from `sphinx_needs.api`. `from_directive(self)` records exactly what a need directive records for its own needs: the line the directive is written on (through `rst_prolog` and includes), its parser line and content offset, and the file and page. It works for a plain docutils `Directive` as well as a `SphinxDirective`. `on_page` is the source `add_need` built. Both classmethods are stable, so an extension that migrates through them moves once, straight to the final spelling (the general constructor stays provisional, below). `add_need` still works and is deprecated with a migration guide. ### What - **`NeedItemSource`**: one frozen dataclass in `sphinx_needs.need_item`, re-exported from `sphinx_needs.api` with the provisional hop types `NeedSourceHop`, `NeedSourcePhysical` and `NeedSourceSemantic`. - Constructor: `NeedItemSource(hops=(), *, lineno=None, lineno_content=None, parser_lineno=None, is_import=False)`. - Named constructors: `NeedItemSource.from_directive(directive, *, first=())` and `NeedItemSource.on_page(docname, *, lineno=None)`. - A need's legacy location keys follow one rule each: `docname` is the placement's document, `lineno` is as given or else the first hop's line, `is_external` and `external_url` come from an external first hop, and the rest are as given. - Stable: its name, `from_directive`, `on_page`, and reading a source through `NeedItemSourceProtocol`. Provisional, like `hops` and the hop types until needs.json carries them: the general constructor and its keywords (`hops`, `lineno`, `lineno_content`, `parser_lineno`, `is_import`), for sphinx-needs' own producers and advanced use; they are the stored legacy facts a later release turns into projections of the hops. `NeedItemSourceProtocol` is unchanged. `test_what_is_stable_of_the_one_class` pins the stable signatures. - **The five classes it replaces** (`NeedItemSourceUnknown`, `-Directive`, `-Service`, `-External`, `-Import`) stay importable until 10.0 as deprecated factories. - Each takes its 9.1.0 keywords and returns a `NeedItemSource` with the location keys that class gave. - Reading the name emits a `DeprecationWarning` at the reader's line. - A type checker sees each as a deprecated function of its keywords, and still reports a misspelt import from `sphinx_needs.need_item`. - The published sphinx-codelinks 2.0 imports one at module level, which is why removing them in 9.x was not an option. - **`add_need` is deprecated**, with removal in 10.0. Each call emits a Python `DeprecationWarning` at the calling line, and the need is created as before. This is not a Sphinx warning, so `-W` builds that use sphinx-codelinks 2.0 or sphinx-test-reports 3.0 keep passing. `generate_need`, `add_external_need` and `ingest_need_record` are not deprecated. - **Migration**: a new "Migrating from `add_need`" section in the API docs. It gives a before/after example, what `from_directive` does to `lineno` (and how `on_page` keeps it), where `on_page` applies, and where each of `add_need`'s arguments goes. As before, a need placed on a page needs a parser state: an event handler or a role can create only an external need. - **Every producer** is on the one class: - the need directive (`from_directive`), `list2need`, `needservice`, `needimport`, `needs_external_needs` and the API's default source; - the needs of sphinx-codelinks and sphinx-test-reports (`on_page`: their page at the directive's line, the source the old class built). `HopsSource` is gone. The incremental build's moved-only comparison (#2414) gets its key from the source itself (`NeedItemSource._without_lines()`). `ENV_DATA_VERSION` 18: the first build after upgrading reads every document again. Nothing else a build shows changes. ### Why Five source classes and a set of hops said the same thing twice. Their constructors took the legacy keywords and the hops, and checked that the two agreed: transition scaffolding from the hops work. `create_need`'s `source` argument was documented as provisional for exactly that reason, so `add_need` could not be deprecated without making extensions migrate twice. Now the class's name is final, and so are the two classmethods an extension writes, `from_directive` and `on_page`. So both things can go: the scaffolding now, and `add_need` at 10.0. ### Tests - **Base-anchored, committed first and green on master**: `tests/test_need_source_legacy_keys.py` snapshots every need's `dict_repr` and every item: - for the source hops project (each producer); - for a new project (`doc_test/doc_need_source_shapes`) whose extension uses every location shape of `add_need`, `ingest_need_record`, `add_external_need`, `create_need` and `generate_need`, and each of the five names with and without hops. Its upgrade-path test rewrites the pickled environment as the five classes stored it, and checks that the rebuild gives a fresh build's needs. The version that test writes is checked against `data.py`'s own record. - **Pinned red first, as strict xfails on master**: `tests/test_need_item_source.py` covers: - each name as a warning factory: the warning at the caller's line, the base's keys and the hops; - the class and its constructor; - the `lineno` rule's discriminating case: a first hop with a line and a different given `lineno`; - one rule per key; - the public names; - `from_directive` for a docutils `Directive` and a `SphinxDirective` under `rst_prolog`; - `add_need`'s warning, attributed to the calling extension's file. `tests/test_creation_call_sites.py` also fails a module of any package that still uses an old name or `HopsSource`. - **Pinned red first, as strict xfails on the branch before their change**: - `on_page` is the source `add_need` builds; - the module `__getattr__` is hidden from type checkers; - the five names' type-checker declarations match the run-time factories; - a need that only moved compares equal through the source's own key, for every kind of source; - that key walks the source's fields, so a field added later is compared (a need differing in it is re-written). - **Also pinned**: - a third party's source takes the incremental build's generic comparison, not the source's own key; - an unknown source's external hop goes first even when hops are given; - `from_directive` keeps the directive's line where the state machine cannot map it; - the directive factory keeps `parser_lineno`. - **Changed pins**: the reprs are the one class's. The agreement-check cases are removed, since nothing is given twice any more. The `add_need` deprecated-key pin now tells its warning apart by message. - **Measured base vs tip** (not committed): - the docs' needs.json is byte-identical (233 needs); - the hops project gives identical dumps serial, `-j 2` and incremental; - a master `_build` rebuilt by this branch is read again and equals a fresh build; - **the published sphinx-codelinks 2.0.1 and sphinx-test-reports 3.0.1** build a project to an equal needs.json on master and on this branch, `-W` included; - ty reports a misspelt import, each old name as deprecated, and keyword misuse as an error. ### Changelog sphinx-needs: ```rst - ✨ ``NeedItemSource``: one class for where a need comes from, public from ``sphinx_needs.api``, with ``NeedItemSource.from_directive(self)`` as the source of a need an extension's directive creates and ``NeedItemSource.on_page(docname, lineno=lineno)`` for a need at a line the caller gives (:issue:`2137`) The source :func:`~sphinx_needs.api.need.create_need` takes is one frozen class, :class:`~sphinx_needs.need_item.NeedItemSource`: the hops the need's text came through, and the location facts the hops cannot give yet (``lineno``, ``lineno_content``, ``parser_lineno``, ``is_import``). Every need sphinx-needs creates has one, and so does every need sphinx-codelinks and sphinx-test-reports create. In a directive, ``NeedItemSource.from_directive(self)`` records what a need directive records: the line the directive is written on (resolved through ``rst_prolog`` and includes), its parser line and content offset, and the file and page it is written in, after any hops given as ``first``; it takes any docutils ``Directive``, not only a ``SphinxDirective``. ``NeedItemSource.on_page(docname, lineno=lineno)`` gives a need its page and the line the caller passes, the source ``add_need`` built: for a need at its directive's parser line, as ``add_need`` took it (sphinx-codelinks' needs sit there), or at a line the directive's data names. The class's name, ``from_directive`` and ``on_page`` are stable, and so is reading a source through ``NeedItemSourceProtocol``; the general constructor and its keywords (``hops``, ``lineno``, ``lineno_content``, ``parser_lineno``, ``is_import``) are provisional, for sphinx-needs' own producers and advanced use, as ``hops`` and the hop types, ``NeedSourceHop``, ``NeedSourcePhysical`` and ``NeedSourceSemantic``, now also importable from ``sphinx_needs.api``, are: they may change without a deprecation, until ``needs.json`` carries them. A need's ``docname``, ``lineno`` and the other location keys are unchanged, and so is ``NeedItemSourceProtocol``; the first build after upgrading reads every document again. See :ref:`api_migrating_add_need`. - ⚠️ ``add_need`` is deprecated, and is removed in 10.0: ``create_need`` replaces it (:issue:`2137`) Each call of :func:`~sphinx_needs.api.need.add_need` is one Python ``DeprecationWarning``, at the line that calls it, naming ``create_need`` and :ref:`api_migrating_add_need`; it creates the need as before. It is not a Sphinx warning, so a build run with ``-W`` (warnings as errors) is not failed by an extension that still calls it, such as sphinx-codelinks 2.0 and sphinx-test-reports 3.0. ``create_need`` takes the need's location as one ``NeedItemSource`` (``NeedItemSource.from_directive(self)`` in a directive), its fields and links as two mappings, and returns the need with its nodes; the migration section maps each of ``add_need``'s arguments. ``generate_need``, ``add_external_need`` and ``ingest_need_record`` are not deprecated. - ⚠️ The five source classes ``NeedItemSourceUnknown``, ``NeedItemSourceDirective``, ``NeedItemSourceService``, ``NeedItemSourceExternal`` and ``NeedItemSourceImport`` are deprecated names of ``sphinx_needs.need_item``, and are removed in 10.0: ``NeedItemSource`` replaces them Each is a function now, taking its class's keywords and returning the ``NeedItemSource`` with the location keys that class gave, and reading the name (``from sphinx_needs.need_item import NeedItemSourceUnknown``) is a ``DeprecationWarning`` at that line. They were not documented API, and what they did beyond their keywords is gone: a source is a ``NeedItemSource``, so ``isinstance``, a ``match`` class pattern or a subclass naming an old name raises ``TypeError`` (a function is no class: test ``isinstance(source, NeedItemSource)``), ``type(source)`` is never one of them, and an annotation ``NeedItemSourceUnknown | None`` evaluated at run time raises too; ``dataclasses.replace`` raises ``TypeError`` (build a new source) and ``dataclasses.asdict`` gives the one class's fields; the attributes ``url`` (``NeedItemSourceExternal``), ``path`` (``NeedItemSourceImport``), ``external_url`` and ``is_external`` (``NeedItemSourceUnknown``) are gone (read ``source.dict_repr``); ``dir(sphinx_needs.need_item)`` and, at run time, ``from sphinx_needs.need_item import *`` no longer include them (a type checker sees each as a deprecated function of its keywords); and no combination of keywords is refused: given ``hops``, the hops decide ``docname``, ``external_url``, ``is_external`` (but ``NeedItemSourceUnknown(is_external=True)`` still puts an external hop first) and, without a ``lineno``, the line, while a ``lineno`` given is the need's, and an ``external_url`` without ``is_external`` is dropped. ``` sphinx-codelinks and sphinx-test-reports: one 🔧 line each. Their needs take their location as `NeedItemSource` and record nothing new. Part of #2137: the deprecation of `add_need`, the step that issue was kept open for. With it, #2137's plan is done apart from the removal itself, which is a 10.0 task. That plan was a new function for every in-tree caller, then `add_need` warning, then removal at the next major. So #2137 can be closed by hand once this merges, unless the 10.0 removal is to be tracked there.

…s far ends as links (#2418) Read by the data-model side before going out of draft: needs.json's `"summary": true` marker and the exchange-contract sentence fit, and nothing collides with its in-flight work. The ubCode side is useblocks/ubcode#4452; this PR merges first. ## What `packages/sphinx-needs` (+ one sentence in `packages/ub-needs-exchange`'s exchange contract, and a reader test). 1. **The stance.** `summary = true` on a link type's `needs_links` entry (any link type, authored or derived) says its links materialise a path, not an edge of the trace. A summary link type is read where it is named (a filter, a needtable column, a rule's `over` / `field`, a needflow's `:link_types:`, a schema rule, a need type's `back_links` count, which counts its computed links too), and a need holds its links and back links as for any link type. It is left out of every view that reads all link types: in sphinx-needs, the needflow default (without `:link_types:` a summary link type is neither drawn nor walked from `:root_id:`, and the `needs_flow_link_types` fallback leaves it out too). A union cannot name one as a member (`needs.derive_invalid`). `summary` beside the deprecated `copy = true` is reported and the copy ignored, so a summary link never leaks into `links`. needs.json's schema marks the link type and its back links `"summary": true`, and ub-needs-exchange passes it through. The stance is one per name: a need type's entry giving it is a `needs.config` finding. 2. **The kinds.** On a link type, `copy` takes `field = "id"` and `over`, and links each need `over` names. `collect` takes `field` (a link type, a derived one included) and `over`, and gives the links `field` holds on each need `over` names. Both take `where` (`c.is_a(...)` included), and both may be declared on need types alone (`[needs.types.<t>.links.<l>]`). Each link appears once by target and part, its condition dropped. A dead link is carried and reported where it is written, rendered as that link type allows, and does not mark the needs below it as having dead links. Such a link type is a summary implicitly. `over` may name a union, read through its members, so a chain can climb a `trace_up`-style union rung by rung. What the link step cannot wait for is refused at load: - an `over` naming back links; - a `collect` of its own link type over a link type another rule computes (it would read itself on every need); - a `field` naming a value field or back links; - a `copy` of a field other than `id`; - a `select` other than `"list"`; - `after`; - `allow_dead_links` beside such a rule. Per need, a path is ordered after the needs it reads; needs reading one another's chain around a ring (through a union too) are `needs.derive_cycle`, and only those. The density guard counts these links; fingerprints leave them out. 3. **Incremental builds.** A `collect` on a link type links needs its need does not name (a chain's far ends), so it is classified as a `links` rule is: in a project with one, every change of a need re-writes every need-relevant page. A `copy` over a link type written in the needs links only the needs its own links name, and stays a read of its links (a `copy` over a computed link type is classified as unnamed too). `ENV_DATA_VERSION` is 19 (17 is #2415's, 18 #2419's): a `copy` / `collect` on a link type, refused by 9.1.0, now computes, and `summary` now changes a needflow's default, with the configuration unchanged. ## Why Closes #2370. The far ends of a chain (the requirements a test result is evidence for, three hops up) were only expressible as a field: strings, with no back links ("the test results that reach me"), no `:need:` rendering, no direct relationship in a graph consumer. The link form adds them. The stance it needs, that such a link is a summary and not a trace edge, applies to authored and other derived link types too. **Bringing it up to master.** The branch was built on the derive branch, since squash-merged as #2384; master was merged in (merge commits, no rebase). The conflicts kept both sides: the changelog's Unreleased bullets, `LinkSchema`'s `severity` (#2395) beside `summary` in the basic-doc snapshot, a path's `select` default, and the corpus (master's `where-follows-inherits` where master has it, this branch's cases after it). Checked against what master gained since: the incremental rules of #2414 (a `collect` on a link type was read as an ordinary `over` and left the old far end's page stale: fixed, red first, with scratch-vs-incremental cases for both kinds); a `[needs.types.<t>.back_links.<l>]` count (#2395) naming a summary or computed link type (allowed, counts every link, pinned); `c.is_a` (#2401) in a path's `where` (pinned in a test and the corpus); a path declared on need types alone and a need type's `summary` (#2400, pinned); the needs.json writer's nullable rule and `""` -> null (#2395, #2400: fields only, beside `summary` on link types), and the needflow change on master (a view category only). Later merges brought in #2415 (hidden-need location) and #2419 (`NeedItemSource`), which took versions 17 and 18, so this branch's bump is 19; every bump stays in `data.py`'s version notes and in the changelog's bump bullet. **Corpus.** `derive_conformance.toml` gains twelve cases on top of master's 92 (104, sha256 `c4721bcc6cb1532b4a39a79c3ba5325c75b006762ca5d25a5e1c78809923ba01`; python_only 2, ubcode_only 1) and an optional `schema.<name>` expectation, documented in the header; master's cases are untouched. ubCode re-vendors it and mirrors the stance and the kinds in useblocks/ubcode#4452, including its own consumers that read all link types: the needflow default, the agent walks, `ubc diff`'s default, the Need Viewer tree and the Cypher schema flag. **Follow-up, not in this PR.** `summary` on a need type's own link entry is a `needs.config` finding today, so an authored link type declared on need types alone cannot be a summary. It can be lifted under the one-name-one-shape rule (#2284): accept it there when every need type declaring the name agrees, and report a disagreement as any other shape mismatch. ## Tests - Red first for both slices and for the incremental fix; two independent reviews reproduced every proof, ruled on the two judgement calls (the refusal of a derived `over` lifted for unions; a call-computed `over` making only that need a cycle, kept) and added mutations of their own, each now killed (the needflow default and fallback, the union refusal, load-time refusals, stratum order, implied summary, dead links, the `forbidden` rendering with both `allow_dead_links` values). After the first merge: mutations of the incremental classification (both directions), of the back links of a computed summary link type, and of `c.is_a` in a `where`, each red. - After the merge, master's unknown-key report (#2388) pins the keys of a link type's entry: `summary` is one now, and its test says so. - `uv run poe lint` 0, `uv run poe typecheck` 0, full `uv run poe test-needs` 4244 passed / 15 skipped (after the #2419 merge), `uv run poe test-ub-needs-exchange` 0 (246 passed), `uv run poe smoke-needs` 0. `docs-needs` warns only on what intersphinx resolves (the inventories are unreachable from the sandbox); none of this PR's references. ## Checklist - [x] I wrote this change myself and have read every line of it; it was not generated automatically from an issue. - [x] I ran the package's tests (`uv run poe test-needs`, `uv run poe test-ub-needs-exchange`) and they pass. - [x] Documentation is updated where behaviour or options change. - [x] The package's `docs/changelog.rst` has an entry under *Unreleased*. - [x] `uv run poe lint` and `uv run poe typecheck` pass.

chrisjsewell added 23 commits October 27, 2026 14:00

👌 The derive rules a need type has are read once per type, not once p…

f841efd

…er need What an incremental build's dependency check reads of each need type's derive rules depends on the type alone, so it is worked out the first time a type is met rather than for every need.

🔧 Incremental: say why 'style' is a field its own need decides

27c79e5

🔀 Merge master into the incremental-rewrite branch (#2406)

37fd14e

create_need takes a source and its fields and links as mappings; it changes nothing the incremental rules read, and ENV_DATA_VERSION stays 16.

🔧 Incremental: re-wrap a long comment line in the rewrite tests

e63062e

github-actions Bot added the pkg: sphinx-needs The sphinx-needs distribution (packages/sphinx-needs): its code, tests and docs label Oct 30, 2026

♻️ The incremental-build module is internal: _incremental.py, as th…

6796f89

…e package's other new internal modules are

chrisjsewell merged commit e4d64d5 into master Oct 30, 2026
45 of 46 checks passed

chrisjsewell deleted the claude/vibrant-allen-o22lej-incremental branch October 30, 2026 12:02

This was referenced Oct 30, 2026

✨ Summary link types, and copy / collect on a link type: a chain's far ends as links #2418

Merged

✨ NeedItemSource: one class for where a need comes from, and add_need is deprecated in favour of create_need #2419

Merged

Navigation Menu

Repository navigation

🐛 An incremental build re-writes every page showing a need that changed - #2414

🐛 An incremental build re-writes every page showing a need that changed#2414
chrisjsewell merged 24 commits into
masterfrom
claude/vibrant-allen-o22lej-incremental

chrisjsewell commented Oct 30, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

chrisjsewell commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Master's newer readers, checked against the rules

For the source-hops work

Why

Tests

Checklist

Uh oh!

codecov Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

chrisjsewell commented Oct 30, 2026 •

edited

Loading

codecov Bot commented Oct 30, 2026 •

edited

Loading