Skip to content

Commit 2db7ae0

Browse files
derek73claude
andcommitted
Check #610's candidate superset at the take, on unjoined pieces
Review found the sweep test read plain names through all of `group`, so the pieces it handed to `trailing_candidates` already carried the joins the maiden take runs before ('de VI Prof.' as one piece where the take sees three). The oracle now wraps `_group._maiden_take` and checks every take of a 37,044-clause grid on the pieces it was actually handed: the clause-free view, `tail_reading` over it, and every piece it takes at or after the cut. 0 misses; the loop #601 shipped misses 432 there, every one the shape-numeral class, which is the recorded negative control. decisions.md#M2 relabels the first grid's figures with their population, and mechanisms.md's Field notes record the trap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent c7f2b58 commit 2db7ae0

3 files changed

Lines changed: 65 additions & 31 deletions

File tree

‎docs/design/decisions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -244,7 +244,7 @@ the fullwidth-colon marker (旧姓:佐藤 arrives as one word; the head-peel q
244244
- 2026-09-27 (Derek), #544 — S2'S COMPANY CLAUSE DOES NOT REACH ACROSS A CLAUSE, recorded as an Accepted boundary rather than repaired. `Jane Doe Jr. nee Smith Ma` keeps maiden 'Smith Ma' and reports it, the clause's own words standing between 'Jr.' and 'Ma', where the clause-less `Jane Doe Jr. Ma` reads suffix 'Jr. Ma'. tests/v2/test_properties.py's clause-agreement walk pins the pairs that differ for exactly this reason as its `anchored_head` class — 810 of its 20,412 pairs, recorded 2026-09-27, 0 with the anchor off. Inside the clause the company reads as it does anywhere: `Jane Doe nee Smith PhD MEng` ends the clause at 'PhD' and reads suffix 'PhD MEng', maiden 'Smith'.
245245

246246
- 2026-10-04 (Derek), #601 — A MARKER COUNTS ONLY BEHIND A SURNAME, AND IT TAKES THE WORDS AFTER IT UP TO THE TRAILING RUN THE NAME WOULD END WITH IF THE CLAUSE WERE NOT WRITTEN. This SUPERSEDES the release machinery of the #533 and #535 entries above (the release question, the per-stop checks, the anchored-head boundary of the #544 entry) and the given-part and suffix-comma reach the 2026-07-03 rule had; the #399, #420, #434 and delimiter entries stand. The trigger was #548, `Dr. nee Smith PhD Prof.`, where 2.3.0 read family 'PhD', maiden 'Smith': a clause whose head was nothing but a title, and a release check asking whether each word it gave up could still read as a post-nominal. #548 was closed into this issue and #602 (S2's run) because both halves of its answer were rules rather than repairs. THE HEAD RULE (Derek): a marker is one only in the name before any comma or the surname part before a family comma, behind at least one name word past the leading title run, and not straight behind an unambiguous suffix word or a connective. Anywhere else it is an ordinary word — in the given part after a family comma (`Doe, Jane nee Smith` reads middle 'nee Smith', 1.4.0's reading) and in a part after a suffix comma (`Smith, John, PhD née Jones` reads suffix 'PhD née Jones', also 1.4.0's). Derek's framing was that a maiden marker always follows a family name, and that a marker following something the parser has decided is a title, suffix, particle or connective is not a marker. THE PARTICLE HALF WAS DROPPED BY MEASUREMENT: prototype variant 1 refused a marker behind particle vocabulary and so declined `Anh Do née Tran` and `Mai Le née Nguyen`, Do, Le and Van being particles that are also the commonest Vietnamese surnames; a particle counts as the surname (`Jane van nee Smith` reads family 'van', maiden 'Smith'). THE BOUND (Derek): the clause takes its words up to the run of post-nominals and titles that the trailing rules read at the end of the name with the clause removed, and the words it gives up take the roles that reading gives them — the TRAILING rule's reading of the clause-free name, not the full parse of it, so a head the full parse would read differently (H1's lone title, a P5 join) cannot change where the clause ends. The first word after the marker is always taken, the marker having announced a name — a lone roman numeral included, so `Jane Smith née V` reads maiden 'V' where the clause-free `Jane Smith V` reads suffix 'V', accepted rather than given a numeral exception (rules.md#M2's Accepted block) — unless it is an unambiguous suffix word, where the marker declines (`Jane Smith nee PhD` reads family 'nee', suffix 'PhD', as before). Before a family comma the clause keeps the 2026-07-03 stop at the first suffix word, since no trailing rule reads that part's end. VARIANTS, prototyped 2026-10-03 behind an environment switch on one tree (the numbers are in #601's comments): v2/v3 grew the clause until the existing release check accepted it and reached 0 invariant violations but inherited the check's caution about particles (`Anh Do geb. de la Vega PhD Prof.` kept 'PhD Prof.' in the maiden name); v4 read the run once over the name as written without binding it, and is the recorded negative control in tests/v2/test_properties.py (350 of the clause grid's 4,482 parses fail under it, measured 2026-10-04); v5/v6 bound that reading as roles but still counted the clause's words among the words to spare, so `John nee Smith ba` read 'ba' as a credential where `John ba` reads a name; v7, the clause-free reading bound as roles, was adopted. Against #535's grids v7 left 0 violations of the M2 invariant where the tree then had 15,586 (grid A, 327,936 texts) and 8,713 (grid B, 173,057), measured 2026-10-03 with an invariant check written to #535's recipe. WHAT THE PROTOTYPE MISSED, found implementing it: the numeral fork. A trailing roman numeral is a suffix only by its shape against the word before it, and v7 read the clause's last word without that test, so `John née Jones Smith VI` kept 'VI' in the maiden name; the take now asks `is_trailing_numeral_suffix` of the last piece, and cases.py's `a_numeral_the_clause_free_name_reads_ends_the_clause` pins it. And the head's title run is measured over the words before the marker, a lone title excepted: measured over the whole segment, `Lord Chancellor née Jones` counted 'Chancellor' as a title and refused the marker. TRIAGE, 2026-10-04: the prototype's 68 failing case rows (40 facade failures among them) were re-pinned as `fix(#601)` with their group named in the note, bar the head-rule bug above; tests/v2/test_parser.py's corpus-append invariant became two-sided (appending ` née X` moves no other field, OR the take records why it declined), with a named set for the suffixes that themselves hold a marker. DEGENERATE INPUT, NO READING DESIGNED FOR (Derek, 2026-10-04: garbage in, garbage out). Two names that read worse by eye moved, and they are recorded here so the move is not mistaken for a regression, not because either reading is wanted: `Berg, Jane van der nee Smith DO` reads middle 'van der nee Smith', family 'Berg' where 2.3.0 read family 'van der Berg', maiden 'Smith DO' — the marker in a given part being a word, the particles no longer trail the given name (P6); and `Jane Doe, PhD née Smith` reads given 'PhD', middle 'née Smith', family 'Jane Doe' where 2.3.0 read given 'Jane', family 'Doe', suffix 'PhD', maiden 'Smith', which 2.3.0 reached only through the clause taking the marker (the comma part opening with a credential is #603's shape). Measured 2026-10-04 against 2.3.0. COST: the take is one function and no release helper survives; the #533 review's comma-head grids could no longer reach a take at all and were replaced by counting heads (tests/v2/test_properties.py's `_maiden_clause_grid`, 4,482 parses).
247-
- 2026-10-04 #610 — THE MAIDEN TAKE'S CANDIDATE WORDS ARE ONE FUNCTION BESIDE THE PEEL, AND SHARING IT FIXED A GAP THE HAND COPY HAD. `_maiden_take` chose the clause words to read its clause-free view over with a filter of its own (`_clause_tail_word` plus a numeral clause), a hand copy of the peel's and the chain's admissions whose contract — a superset of what `tail_reading` takes — nothing stated or tested. It is now `_pieces.trailing_candidates`, written beside `peel_trailing`, and tests/v2/pipeline/test_pieces.py pins the superset over a generated grid (37,044 parses through `group`). The grid found the copy wrong twice. The numeral fork reads a shape-only roman numeral (`VI`, in no wordlist) as the WALK's last piece, and two things leave it last without standing at the end of the name: a period-marked title behind it, which the chain takes first, and the merged `Ph. D.`, which group flags and the walk never holds. The copy admitted the numeral only as the clause's last word, so `Jane Doe nee Smith VI Prof.` kept maiden 'Smith VI' where `John Smith VI Prof.` reads suffix 'VI', and `Jane Doe nee Smith VI Ph. D.` the same way. Both now end the clause at the numeral, as rules.md#M2's statement already said they should. Recorded negative control: the loop as #601 shipped it misses 288 texts of the grid, every one this shape; a first draft of the fix covered the title half and missed the `Ph. D.` half, 111 texts. Against master 8a8459a8 over the 95,813-text equivalence grid the change moved 10 readings, all of this class. The same issue lifted P6's particle-tail walk into `_pieces.particle_tail`, read off roles, so post_rules' attachment and assign's given-part credential run (#602) ask one walk; that half moved none of the 95,813 readings.
247+
- 2026-10-04 #610 — THE MAIDEN TAKE'S CANDIDATE WORDS ARE ONE FUNCTION BESIDE THE PEEL, AND SHARING IT FIXED A GAP THE HAND COPY HAD. `_maiden_take` chose the clause words to read its clause-free view over with a filter of its own (`_clause_tail_word` plus a numeral clause), a hand copy of the peel's and the chain's admissions whose contract — a superset of what `tail_reading` takes — nothing stated or tested. It is now `_pieces.trailing_candidates`, written beside `peel_trailing`, and tests/v2/pipeline/test_pieces.py pins the superset at the take itself, on the unjoined pieces the take is handed, over a generated grid (37,044 clauses through `group`). The grid found the copy wrong twice. The numeral fork reads a shape-only roman numeral (`VI`, in no wordlist) as the WALK's last piece, and two things leave it last without standing at the end of the name: a period-marked title behind it, which the chain takes first, and the merged `Ph. D.`, which group flags and the walk never holds. The copy admitted the numeral only as the clause's last word, so `Jane Doe nee Smith VI Prof.` kept maiden 'Smith VI' where `John Smith VI Prof.` reads suffix 'VI', and `Jane Doe nee Smith VI Ph. D.` the same way. Both now end the clause at the numeral, as rules.md#M2's statement already said they should. Recorded negative control: the loop as #601 shipped it misses 432 of that grid's clauses, every one this shape. The test's first version read plain names through all of `group`, on pieces the joins had already merged, which review caught: there the old loop missed 288 names, and a first draft of the fix, covering the title half only, still missed 111 — the `Ph. D.` half, which that grid is what found. Against master 8a8459a8 over the 95,813-text equivalence grid the change moved 10 readings, all of this class. The same issue lifted P6's particle-tail walk into `_pieces.particle_tail`, read off roles, so post_rules' attachment and assign's given-part credential run (#602) ask one walk; that half moved none of the 95,813 readings.
248248

249249
### N1 — the default delimiter pairs
250250

‎docs/design/mechanisms.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,7 @@ Problem shape. A test pins an ordering, a sort, a dedup or a partition, and its
186186

187187
### Field notes — the traps themselves
188188

189+
- Test a function INSIDE a stage on the inputs its caller hands it, not on the stage's finished output. #610's first sweep test built pieces by running `group` whole and handed them to the maiden take's candidate function, but the take runs before any join, so the test read `de VI Prof.` as one piece where production hands it three: a guard checking a structure production never passes in, green for the wrong reason. Review moved the oracle to the take itself by wrapping it (tests/v2/pipeline/test_pieces.py), and recorded the old loop's misses there as the negative control (432), which is what shows the moved harness can still see a difference.
189190
- Enumerate the rules that BUILD a structure; do not recall them. #395's unit walk was written three times in one PR — P2's chain missing, then the conjunction and bound-given branches absorbing one token where the particle branch absorbed a unit, then the suffix stop — and each miss came from listing the joining rules from memory instead of reading them out of rules.md. Both later misses reproduced the very defect the first fix had just removed, mirrored.
190191
- Prefer monkeypatching a module-level helper to re-exec'ing a module: a caller looks a module global up at CALL time, so replacing `_units` or `_name_positions` reaches it and none of the traps below apply. Re-exec is only needed for logic INLINE in a function, and "the thing I want to mutate is not a module-level helper" is a hint that it could be one.
191192
- Re-exec'ing a module redefines the classes that module DEFINES, not the ones it imports — the exec re-runs its own import statements, which resolve through sys.modules. So `is` comparisons break selectively: re-exec'ing `_group.py` yields a second `BoundJoin` (defined there) while `Structure` (imported) stays identical, and `bound_join is not BoundJoin.DISABLED` is then always true, joining a family comma's own segment and reporting 3,150 phantom movers. Rebind what the module defines (`mod.__dict__["BoundJoin"] = real.BoundJoin`) and assert the identity rather than assuming it either way.

‎tests/v2/pipeline/test_pieces.py‎

Lines changed: 63 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
"""
88
import dataclasses
99
import itertools
10-
from collections.abc import Sequence, Set
10+
from collections.abc import Callable, Sequence, Set
1111

1212
import pytest
1313

@@ -16,6 +16,7 @@
1616
from nameparser._pipeline import STAGES
1717
from nameparser._pipeline._assign import assign
1818
from nameparser._pipeline._classify import classify
19+
from nameparser._pipeline import _group
1920
from nameparser._pipeline._group import group
2021
from nameparser._pipeline._pieces import (
2122
_anchors, _numeral_behind_the_initial_veto, anchor_in_reach,
@@ -26,7 +27,7 @@
2627
)
2728
from nameparser._pipeline._segment import segment
2829
from nameparser._pipeline._state import (
29-
AMBIGUOUS_ACRONYM_TAG, ParseState, WorkToken,
30+
AMBIGUOUS_ACRONYM_TAG, ParseState, PendingAmbiguity, WorkToken,
3031
)
3132
from nameparser._pipeline._tokenize import tokenize
3233
from nameparser._pipeline._vocab import is_one_case, is_title_shaped, tag_marker_runs
@@ -890,56 +891,88 @@ def test_a_title_inside_the_given_parts_run_is_a_title() -> None:
890891

891892
# #610: `trailing_candidates` is the maiden take's choice of which
892893
# clause words to read the clause-free view over, and its contract is
893-
# a SUPERSET of what `tail_reading` takes. The grid reads plain names,
894-
# where the candidates' cut must fall at or before every piece the tail
895-
# reading takes (the first name piece aside: in a clause that is the
896-
# word after the marker, which the take always keeps).
894+
# a SUPERSET of what `tail_reading` takes from that view. Checked AT
895+
# THE TAKE, on the pieces the take is handed -- before any join, which
896+
# is where it runs -- by wrapping `_group._maiden_take` and the
897+
# candidates call inside it: the clause-free name is the head plus every
898+
# clause word after the first (which the take always keeps), and every
899+
# piece the tail reading takes from it must fall at or after the cut.
897900
_CANDIDATE_HEADS = ("Jane Doe", "J.", "Jane van der Berg", "Mai Le",
898901
"abdul Berg", "John")
899902
_CANDIDATE_WORDS = ("Smith", "VI", "V", "III", "MA", "Ma", "PhD", "Jr.",
900903
"Prof.", "Dr.", "King.", "do", "DO", "de", "X.Y.Z.",
901904
"Ph. D.", "i", "Jones")
902905

903906

904-
def _candidate_misses() -> list[str]:
905-
misses = []
907+
def _candidate_misses(monkeypatch: pytest.MonkeyPatch,
908+
candidates: Callable[..., int] = trailing_candidates,
909+
) -> list[str]:
910+
misses: list[str] = []
911+
seen: list[tuple[int, int]] = []
912+
real_take = _group._maiden_take
913+
914+
def cut(lo: int, pieces: Sequence[Sequence[int]],
915+
ptags: Sequence[Set[str]],
916+
tokens: Sequence[WorkToken]) -> int:
917+
c = candidates(lo, pieces, ptags, tokens)
918+
seen.append((lo, c))
919+
return c
920+
921+
def take(pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]],
922+
tokens: Sequence[WorkToken], one_case: bool | None,
923+
site: _group.ClauseSite,
924+
ambiguities: list[PendingAmbiguity],
925+
) -> _group.MaidenIndices | None:
926+
seen.clear()
927+
answer = real_take(pieces, ptags, tokens, one_case, site,
928+
ambiguities)
929+
for lo, c in seen:
930+
m = next(v for v in range(1, len(pieces))
931+
if _group._is_maiden_marker_piece(pieces[v], tokens))
932+
left = list(range(m)) + list(range(lo + 1, len(pieces)))
933+
view = [pieces[q] for q in left]
934+
vtags = [ptags[q] for q in left]
935+
rest = peel_walk(leading_titles(view, vtags, tokens), vtags)
936+
kept, chained, peel = tail_reading(rest, view, vtags, tokens,
937+
one_case)
938+
taken = {left[q] for q in (*kept[peel.names:], *chained)
939+
if q >= m}
940+
if any(j < c for j in taken):
941+
misses.append(" ".join(tokens[i].text for p in pieces
942+
for i in p))
943+
return answer
944+
945+
monkeypatch.setattr(_group, "trailing_candidates", cut)
946+
monkeypatch.setattr(_group, "_maiden_take", take)
906947
for head in _CANDIDATE_HEADS:
907948
for n in (1, 2, 3):
908949
for tail in itertools.product(_CANDIDATE_WORDS, repeat=n):
909-
text = f"{head} {' '.join(tail)}"
910-
state = _through_group(text)
911-
pieces, ptags = state.pieces[0], state.piece_tags[0]
912-
tokens = state.tokens
913-
lead = leading_titles(pieces, ptags, tokens)
914-
rest = peel_walk(lead, ptags)
915-
if not rest:
916-
continue
917-
kept, chained, peel = tail_reading(
918-
rest, pieces, ptags, tokens, state.one_case)
919-
taken = (set(kept[peel.names:]) | set(chained)) - {rest[0]}
920-
c = trailing_candidates(rest[0], pieces, ptags, tokens)
921-
if any(j < c for j in taken):
922-
misses.append(text)
950+
_through_group(f"{head} nee Smith {' '.join(tail)}")
923951
return misses
924952

925953

926-
def test_the_trailing_candidates_cover_what_the_tail_reading_takes() -> None:
954+
def test_the_trailing_candidates_cover_what_the_tail_reading_takes(
955+
monkeypatch: pytest.MonkeyPatch) -> None:
927956
"""M2's take ends the clause at the trailing run the clause-free
928957
name reads, which it can only find if every word that run might
929-
hold is in its view. 6 heads x 18 words to a
930-
tail of three, 37,044 parses through `group`, about 2.4s on py3.11
958+
hold is in its view. 6 heads x 18 words to a tail of three after
959+
'nee Smith', 37,044 parses through `group`, about 2.6s on py3.11
931960
(measured 2026-10-04).
932961
933962
RECORDED NEGATIVE CONTROL, measured 2026-10-04 on this grid: the
934963
candidate loop as #601 shipped it (master 8a8459a8), which admitted
935964
a roman numeral by its shape only as the clause's LAST word, misses
936-
288 texts -- every one a numeral in no wordlist ('VI') with a
937-
period-marked title or the merged 'Ph. D.' behind it. The chain
965+
432 texts -- every one a numeral in no wordlist ('VI') with
966+
a period-marked title or the merged 'Ph. D.' behind it. The chain
938967
takes the title first, and the walk never holds the flagged
939968
'Ph. D.', so either way the numeral is the walk's last piece and
940969
the fork reads it: 'Jane Doe nee Smith VI Prof.' kept maiden 'Smith
941970
VI' where 'John Smith VI Prof.' reads suffix 'VI'. A first draft of
942-
the fix covered the title and still missed the 'Ph. D.' half, 111
943-
texts, which this grid is what found.
971+
the fix covered the title half and missed the 'Ph. D.' half.
972+
973+
A first version of this test read plain names through all of
974+
`group`, so its pieces had the joins applied that the take runs
975+
before ('de VI Prof.' arrived as one piece); review moved the
976+
oracle to the take itself.
944977
"""
945-
assert _candidate_misses() == []
978+
assert _candidate_misses(monkeypatch) == []

0 commit comments

Comments
 (0)